AWS S3 WebTx Plugin for Log Shipper

This document explains how to configure your AWS S3 WebTx integration with the Log Shipper module of the Netskope Cloud Exchange platform. This integration allows pushing web transactions data into AWS S3 buckets. The AWS S3 WebTx plugin support is:

Event Support	No
Alert Support	No
WebTx Support	Yes

Prerequisites

A Netskope Tenant (or multiple, for example, production and development/test instances).
A Netskope Cloud Exchange tenant with the Log Shipper module already configured.
AWS S3 credentials with create/read/write buckets permissions. Obtain your AWS S3 Access key ID and Secret Access Key with permission to read/write on specific buckets or permission to create buckets before proceeding.

Note

Verify your bucket permissions are secure and not set up for open public access. Only allow access to your cloud storage instance from your Cloud Exchange Host and any other addresses that need access.

Workflow

Create a bucket, an IAM policy and user, and get your credentials.
Configure the AWS S3 WebTx plugin.
Configure Log Shipper Business Rules for AWS S3 WebTx.
Configure Log Shipper SIEM mappings for AWS S3 WebTx.
Validate the AWS S3 WebTx plugin.

Create a Bucket, an IAM Policy and User, and Get your Credentials

Create an AWS S3 Bucket

To create bucket and set required policies, go to your AWS console and log in.
From All Services, select S3.
Click Create Bucket.
Enter a Bucket Name, scroll to the bottom, and click Create Bucket.
Search for your bucket and click on the bucket name to open it.
Click on Permission to open the permission tab and to set a policy.
Click Edit. Under Block public access (bucket settings), uncheck all checkboxes and click Save Changes. When prompted for a confirmation, confirm it, and then click Confirm.
Under the Permissions tab, click Edit in the Bucket Policy section.
Click Policy Generator.
Select S3 Bucket Policy as policy type for Step 1, and Add Statement details for Step 2, and then click Generate Policy.
- Select Type of Policy: S3 Bucket Policy
- Effect: Allow
- Principal: <user-arn>
- Actions:
  1. GetBucketAcl
  2. GetBucketPolicy
- ARN: arn:aws:s3:::<bucket-name>
- Click Add Statement.
- Select Type of Policy: S3 Bucket Policy
- Effect: Allow
- Principal: <user-arn>
- Actions:
  1. PutObject
  2. PutObjectAcl
- ARN: arn:aws:s3:::<bucket-name>/*. Be sure to add /* after the second bucket name.
- Click Add Statement
- Click Generate Policy.

Add this policy to this textbox.

{
    "Id": "<policy ID>",
    "Version": "<version>",
    "Statement": [
        {
            "Sid": "<statement ID>",
            "Action": [
                "s3:GetBucketAcl",
                "s3:GetBucketPolicy"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::<bucket-name>",
            "Principal": {
                "AWS": [
                    "<user-arn>"
                ]
            }
        },
        {
            "Sid": "<statement ID>",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::<bucket-name>/*",
            "Principal": {
                "AWS": [
                    "<user-arn>"
                ]
            }
        }
    ]
}

Scroll to the bottom and click Save Changes.

Create an IAM Policy

Go to Policy Generator , select IAM Policy as the policy type, and then click Generate Policy.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: Amazon S3
- Actions:
  1. ListAllMyBuckets
- ARN: *
- Click Add Statement.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: Amazon S3
- Actions:
  1. GetBucketAcl
  2. GetBucketPolicy
  3. GetBucketLocation
- ARN: arn:aws:s3:::<bucket-name>
- Click Add Statement.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: Amazon S3
- Actions:
  1. PutObject
  2. PutObjectAcl
- ARN: arn:aws:s3:::<bucket-name>/*
- Click Add Statement.
- Click Generate Policy.

Create the policy required IAM, as shown below. Be sure to add /* after the second bucket name.

{
	"Id": "<policy ID>",
	"Version": "<version>",
	"Statement": [
		{
			"Sid": "<statement ID>",
			"Effect": "Allow",
			"Action": "s3:ListAllMyBuckets",
			"Resource": "*"
		},
		{
			"Sid": "<statement ID>",
			"Effect": "Allow",
			"Action": [
				"s3:GetBucketAcl",
				"s3:GetBucketPolicy",
                           "S3:GetBucketLocation"
			],
			"Resource": "arn:aws:s3:::<bucket-name>"
		},
		{
			"Sid": "<statement ID>",
			"Effect": "Allow",
			"Action": [
				"s3:PutObject",
				"s3:PutObjectAcl"
			],
			"Resource": "arn:aws:s3:::<bucket-name>/*"
		}
	]
}

Go to AWS Console and select IAM from All Services. Click Policies in the left panel, and then click Create Policy.
Copy that policy to the JSON tab. Click Next:Tags, and then click Next:Review.
Enter a name and click Create Policy.

Create an IAM User

To create an IAM User, and get the Access ID and Secret Access Key, go to the AWS Console and select IAM from All Services.
Click Users in the left panel.
Click Add Users.
Enter a user name and click Next:Permissions.
Click Attach existing policies directly (the policy created earlier) and click Next.
Review your choices and click Create User.
Search for a user and click on it.
Click Security Credentials and scroll down to Access Keys.
Click Create access key, select Other, and then click Next.
Add a description and click Create access key.
Copy the Access Key ID and Secret Access Key.

To Set a Policy from the AWS CLI

The account that owns the access keys should have the below policy.

To set a bucket policy from the AWS CLI

aws s3api put-bucket-policy --bucket MyBucket --policy file://policy.json

policy.json:

{
    "Policy": {
        "Version": "<version>",
        "Statement": [
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<user-arn>"
                },
                "Action": [
                    "s3:GetBucketAcl",
                    "s3:GetBucketPolicy"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>"
            },
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<user-arn>"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>/*"
            }
        ]
    }
}

To set an IAM policy from the AWS CLI

aws iam put-user-policy --user-name <username> --policy-name <policy_name> --policy-document file://iam_policy.json
iam_policy.json:

{
    "Policy": {
        "Version": "<version>",
        "Statement": [
            		     {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Action": "s3:ListAllMyBuckets",
                "Resource": "*"
            },
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Action": [
                    "s3:GetBucketAcl",
                    "s3:GetBucketLocation",
                    "s3:GetBucketPolicy"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>"
            },
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>/*"
            }
        ]
    }
}

Configure the AWS S3 WebTX Plugin

In Cloud Exchange, go to Settings > Plugins.
Search for and select the AWS S3 WebTx box to open the plugin creation pages.
Enter a Configuration Name.
Click Next.
Enter AWS Access Key ID: AWS S3 access key id for AWS S3 authentication.
Enter AWS Secret Access Key: AWS S3 secret access key for AWS S3 authentication.
Select Region Name: Region name to get AWS S3 Bucket from.
Enter Bucket Name: Bucket Name in which the data object will be stored.
Enter Object prefix: It will be added to the Object key name while ingesting web transactions to AWS S3 bucket.
Enter Maximum File Size: Maximum size of WebTx data object to be stored in the bucket. (Value should be between 0 to 100.)
Enter Maximum Duration: Maximum duration after which the WebTx data object should be stored in the bucket.
Note
Maximum File Size and Maximum Duration whichever hits the first, data will be exported according to that.
Click Save.

Configure Log Shipper SIEM Mappings for AWS S3 WebTx

Go to Log Shipper > SIEM Mappings and click Add SIEM Mapping.
For source, select the Netskope WebTx configuration.
For destination, select the AWS S3 WebTx configuration.
Click Save.

Validate the AWS S3 WebTxPlugin

To validate the plugin workflow, you can check from Netskope Cloud Exchange and from AWS CLI.

To validate from Netskope Cloud Exchange,

Go to Logging.

To validate from the AWS CLI.

Install AWS CLI in your machine.
(Reference: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
Configure AWS CLI in your machine.
(Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)
Use these commands to view buckets and objects.
To list all the buckets.
```
aws s3 ls
```
To list all the objects in a bucket.
```
aws s3 ls s3://<bucket_name>
```

To download an object from a bucket to a local machine.

aws s3 cp s3://<bucket_name>/<object_path> <local_path>

Troubleshooting the AWS S3 WebTx Plugin

If you receive the Provided bucket name already exists at a different region. Please try with a different name or use the correct region. error while upgrading the plugin, follow the below steps.

Click Skip.
Go to the Configured plugin.
Click Edit.
Select the actual region in which the bucket is present.
Click Save.
When successful, save and enable the plugin.

In this section:

Netskope Help