Skip to main content

Netskope Help

Add Custom User Attributes

Use the REST API to upload custom user attributes like org unit, location, country, manager, and so on, to enrich Skope IT events. This is an alternate method to using Directory Importer (formerly know as Active Directory {AD} Importer). While Directory Importer fetches attributes directly from a user directory, like AD or other directory, this method allows you to use data from other sources, like an HR tool or any other IAM. The data from such tools is often more accurate than data from an AD.

The REST API enables you to upload a CSV file with all the custom user attributes directly to the Netskope UI.

Important CSV File Properties

The following are some important factors to consider when using CSV file to import users.

  • The file uploaded must be in CSV format with a header row that defines the custom attribute fields.

  • The names in the header fields are used for the field names displayed in Skope IT events.

  • The first field must be the email ID, which must map to users' email address.

  • Field names should not have spaces or double quotes.

  • The custom attributes can be in double quotes, and must be inside double quotes if they have spaces or a comma in the value.

  • The first row of the CSV must contain field names which will be used as custom attribute field name.

  • Maximum number of custom attributes supported is 15.

  • There is no limit on the length of the value for the attributes, but it is recommended that the length be kept as short as possible. Maximum recommended length for the values is 40.

  • If the header is in uppercase or mixed case, it will be converted to lowercase in accordance with the other Netskope attributes.

  • If there are duplicate key records, the first one will be the one considered. The duplicates are resolved based on the ‘email_id’ field. The email_id field is considered as the unique key field.

  • Extensions allowed for the files are .csv and .txt.

  • If the CSV file has more than 1000 records, the import script automatically splits the file into 1000 row chunks. The script to add custom user attributes is located at the end of this topic.

Example of a CSV File

EMAIL_ID,USER_ID,COMPANY_CODE,ORGANIZATION_UNIT,COUNTRY_CODE,LOCATION,USER_TYPE
2"user1@org.com","user1",143,"NorCal","US","SC","Internal"
3"user2@org.com","user2",143,"NorCal","US","SC","Internal"
4"user3@org.com","user3",143,"SoCal","US","SD","Internal"
Using the Script

Follow the procedure below to import users using the script.

  1. Build your CSV file following the guidelines stated and save it to a local directory. XY

  2. Get your REST API token from the Netskope UI (Settings > Tools > Rest API).

  3. Copy the script to a local directory.

  4. Run the script as follows:

    For fresh import, run the script in two steps, 1 - Delete existing entries & 2 - Import new entries.

    Delete Existing Entries

    ./ns_send_adsync.sh -f oldusers.csv -u "https://addon-um-automation1.inskope.com/adsync/z99W4kf7fhh5FgjexPl?token=68fcc5b6edfd88e3c833248a166d6f6d" -o "delete"

    Import New Entries

    ./ns_send_adsync.sh -f newusers.csv -u "https://addon-um-automation1.inskope.com/adsync/z99W4kf7fhh5FgjexPl?token=68fcc5b6edfd88e3c833248a166d6f6d"

    Note

    It's important you see the message "Data successfully Uploaded to Netskope". Otherwise, the upload was not successful. You can parse the log file and verify it in the Skope IT events.

Sample Usage
./ns_send_adsync.sh -f sample1.csv -u 
"https://addon-um-automation1.inskope.com/adsync/z99W4kf7fhh5FgjexPl?token=68fcc5b6edfd88e3c833248a166d6f6d" -o "delete"

Sample Output

BaseURL  : https://addon-um-automation1.inskope.com
Orgkey   : z99W4kf7fhh5FgjexPl
secureUPN: 0
UploadFilesize=3
MAXCHUNKSIZE=1000
constructing and sending adsync request for : sample1.csv
{"config":{"secureUPN":"0"}, "users":[{"action": "delete","email": "user1@org.com","upn": "user1@org.com","enabled":true,"groups":[],"custom_attributes": {"EMAIL_ID": "user1@org.com","USER_ID": "user1","COMPANY_CODE": "143","ORGANIZATION_UNIT": "NorCal","COUNTRY_CODE": "US","LOCATION": "SC","USER_TYPE": "Internal"}},{"action": "delete","email": "user2@org.com","upn": "user2@org.com","enabled":true,"groups":[],"custom_attributes": {"EMAIL_ID": "user2@org.com","USER_ID": "user2","COMPANY_CODE": "143","ORGANIZATION_UNIT": "NorCal,Eternity","COUNTRY_CODE": "US","LOCATION": "SC","USER_TYPE": "Internal"}},{"action": "delete","email": "user3@org.com","upn": "user3@org.com","enabled":true,"groups":[],"custom_attributes": {"EMAIL_ID": "user3@org.com","USER_ID": "user3","COMPANY_CODE": "143","ORGANIZATION_UNIT": "SoCal","COUNTRY_CODE": "US","LOCATION": "SD","USER_TYPE": "Internal"}}], "schema":["EMAIL_ID","USER_ID","COMPANY_CODE","ORGANIZATION_UNIT","COUNTRY_CODE","LOCATION","USER_TYPE"]}
{"status":"success","msg":"Successfully submitted data ","errors":[],"warnings":[]}role
Script Listing
#!/bin/bash 
# ***********************************************************
# Example Bash script to upload csv files for custom user
# attributes
# Copyright: Copyright (c) 2021, Netskope, Inc
# ***********************************************************
ADSYNCURL=
CSVFILE=
OPERATION="patch"

function usage
{
    echo "usage: $0 -u <uiendpoint> -f <path of the csv file to upload> [-o <\"delete\"|\"create\">] [-h]"
}

function get_schema() {
    csvFile=$1
    head -1 $csvfile|gawk '
BEGIN {
    FPAT = "([^,]+)|(\"[^\"]+\")"
    printf("[")
}
{
    if (NR==1) {
        nfields=NF
        for (i=1; i<=NF; i++) {
            gsub("\"","",$i)
            printf("\"%s\"",tolower($i))
            if (i<NF) {
                printf(",")
            }
        }        
    }
}
END {
    printf("]")
}' 
}

function construct_user_request() {
    action=$1
    csvFile=$2
    gawk '
BEGIN {
    FPAT = "([^,]+)|(\"[^\"]+\")"
    count=0
    nfields=0
    printf("[")
}
{
    if (NR==1) {
        nfields=NF
        for (i=1; i <=NF; i++) {
            fileds[i]=tolower($i)
        }        
    }
    if (NR > 2) {
        printf(",")
    }
    if (NR>1) {
        if(NF==nfields) {
            gsub("\"","",$1)
            printf("{")
            printf("\"action\": \"'$action'\",")
            printf("\"email\": \"%s\",",$1)
            printf("\"custom_attributes\": {")
            for (i=1; i <=NF; i++) {
                gsub("\"","",$i)
                printf("\"%s\": \"%s\"",fileds[i],$i)
                if (i < NF) {
                    printf(",")
                }
            }        
            printf("}")
            printf("}")
        } 
        ++count
    }
}
END {
  printf("]")
}' ${csvFile}
}

BASEURL=
ORGKEY=
TOKEN=
secureUPN="0"
function get_secureUPN() {
    BASEURL=${ADSYNCURL%/adsync*}
    echo "BaseURL  : $BASEURL"
    ORGKEY=${ADSYNCURL#*/adsync/}
    ORGKEY=${ORGKEY%\?*}
    echo "Orgkey   : $ORGKEY"
    TOKEN=${ADSYNCURL#*token=}
    TOKEN=${TOKEN%&*}
    response=`curl -X GET --silent -k "$BASEURL/adconfig?orgkey=$ORGKEY&capability=1"`
    secureUPN=`echo $response|grep '"secureUPN"\s*:\s*"[0-2]"' -o|awk -F':' '{if (NF<2){print "\"0\""}else{print $2 }} '`
    secureUPN=${secureUPN#*\"}
    secureUPN=${secureUPN%\"*}
    if [ -z "${secureUPN}" ]; then
        echo "NOTICE : Couldn't get secureUPN config from adconfig, Setting it to \"0\"  " 
        secureUPN="0"
    fi
    echo "secureUPN : $secureUPN"
}

function construct_and_send_request() {
    action=$1
    csvFile=$2
    #construct_user_request "create" $CSVFILE
    users=`cat $csvFile|construct_user_request $action`
    schema=`cat $csvFile|get_schema`
    request="{\"config\":{\"secureUPN\":\"$secureUPN\"}, \"users\":$users, \"schema\":$schema}"
    echo $request
    ADSYNC_RESPONSE=`curl -k --silent -X POST \
                -H 'Content-Type:application/json' \
                -H 'UserAgent:Netskope Adapters -v78.0.0.111' \
                -H 'Accept:application/netskope.adsync.v8' \
                -d "$(echo $request)" \
                -k "$ADSYNCURL"`
    echo $ADSYNC_RESPONSE
}


function uploadFile() {
    UPLOADFILE="$1"
    OPERATION=$2
    declare -i UPLOADFILESIZE=`tail -n +2 ${UPLOADFILE}|wc -l| awk '{print $1}'`
    echo "UploadFilesize=$UPLOADFILESIZE"
    # MAXNUMOFUSERSPERREQ
    declare -i MAXUPLOADSIZE=1000
    echo "MAXCHUNKSIZE=$MAXUPLOADSIZE"
    if [ ${UPLOADFILESIZE} -ge ${MAXUPLOADSIZE} ]; then
        UPLOADCHUNKFILE=`echo chunked_${UPLOADFILE}`
        echo "CSV file contains more than $MAXUPLOADSIZE records. splitting......"
        echo "Chunked File that will be uploaded is $UPLOADCHUNKFILE"
        if [ -d "./nsuploadtmp" ]; then
            echo 'Deleting the temporary nsuploadtmp directory'
            rm -rf ./nsuploadtmp
        fi
        # create a working dir
        mkdir -p nsuploadtmp
        tail -n +2 $UPLOADFILE|split -l $MAXUPLOADSIZE - --filter='sh -c "{ head -n 1 '$UPLOADFILE'; cat; } > $FILE"' nsuploadtmp/ns_uacsv_
        FILELIST=`ls nsuploadtmp/ns_uacsv_*`
        for f in $FILELIST
        do
            echo "constructing and sending adsync request for : $f"
            construct_and_send_request $OPERATION $f
        done
    else
        echo "constructing and sending adsync request for : $UPLOADFILE"
        construct_and_send_request $OPERATION $UPLOADFILE
    fi
}

OPERATION="patch"
while [ "$1" != "" ]; do
    case $1 in
        -u | --uiendpoint ) shift
                ADSYNCURL=$1
                 ;;
        -f | --csvfile )    shift
                CSVFILE=$1
        ;;
        -o | --operation ) shift
                OPERATION=$1
        ;;
        -h | --help ) usage
                      exit 1
                                ;;
        * )           usage
                      exit 2
    esac
    shift
done

get_secureUPN $ADSYNCURL
uploadFile $CSVFILE $OPERATION
In this section: