Data restrictions

The NASA Science Cloud supports FISMA Low by default. Some specifically designated accounts support FISMA Moderate.

Do not transfer, upload, store, process, or publish the following in the NASA Science Cloud:

• Personally Identifiable Information (PII)
• International Traffic in Arms Regulations (ITAR)-controlled data
• Export Administration Regulations (EAR)-controlled data

These restrictions also apply to FISMA Moderate accounts. A Moderate designation does not authorize PII, ITAR-controlled data, or EAR-controlled data.

1. Overview

This page describes methods for transferring data into an existing NASA Science Cloud account from locations such as:

• NASA on-premises systems
• Scientific instruments and processing environments
• Managed workstations or servers
• External research institutions
• Partner systems
• Other approved cloud environments

All destination accounts discussed on this page are assumed to be existing NASA Science Cloud accounts. Before starting a transfer, the user must confirm that the data is appropriate for the destination account’s security categorization.

This page focuses on:

• Amazon S3 command-line and application transfers
• AWS DataSync
• AWS Transfer Family
• Amazon S3 Transfer Acceleration
• AWS Storage Gateway
• Offline transfer options
• Comparable Microsoft Azure services
• Security, integrity, performance, and cost best practices

This page does not cover:

• Transfers out of the NASA Science Cloud
• Public release or publication approval
• Enterprise network architecture
• Transfers of PII
• Transfers of ITAR-controlled data
• Transfers of EAR-controlled data
• Exceptions to NASA security requirements

2. Determine Whether the Data Is Eligible

Determine the applicable FISMA impact level and check for prohibited content before selecting a transfer method.

FIPS Publication 199 defines Low, Moderate, and High impact levels based on the potential effect of a loss of confidentiality, integrity, or availability.[1]

Supported data categories

FISMA Low does not mean public.

Data hosted in a FISMA Low environment may still require authentication, access restrictions, embargo controls, licensing restrictions, or other protections.

FISMA Moderate data

Before transferring FISMA Moderate data, confirm all of the following:

• The destination account is specifically designated to support FISMA Moderate.
• The selected storage location is part of the Moderate environment.
• The transfer uses the account’s established identity, encryption, logging, and monitoring controls.
• The dataset does not contain PII.
• The dataset does not contain ITAR-controlled data.
• The dataset does not contain EAR-controlled data.
• Any project-specific Moderate handling procedures are followed.

Do not temporarily stage Moderate data in a standard FISMA Low account.

Mixed datasets

Treat a dataset according to its highest applicable impact level or restriction.

For example, a scientific dataset containing a small amount of PII must not be uploaded, even if the remainder of the dataset is public or FISMA Low. Remove the prohibited content and complete any required review before transferring the dataset.

Do not upload data to a landing or quarantine location for the purpose of determining whether it contains prohibited information. Perform that review before transfer.

3. Pre-Transfer Eligibility Check

Complete this check before initiating a transfer.

4. Choose a Transfer Method

The appropriate method depends on:

• Total data volume
• Number of files or objects
• Average and maximum file size
• Available bandwidth
• Transfer deadline
• Transfer frequency
• Source storage protocol
• Whether source data will continue changing
• Integrity-verification requirements
• Operational and commercial cost

AWS selection guide

The transfer method does not change the data eligibility rules. No transfer service authorizes PII, ITAR-controlled data, EAR-controlled data, or data above the destination account’s supported impact level.

5. AWS Transfer Methods

5.1 AWS Command Line Interface or AWS SDK

The AWS Command Line Interface, AWS SDKs, and Amazon S3 APIs can upload files and objects directly to Amazon S3. AWS CLI high-level S3 commands manage multipart behavior for large objects.[2][3]

Recommended for

• Small or moderate datasets
• One-time or occasional transfers
• Automated pipelines
• Application-generated data
• Sources that already integrate with AWS APIs

Best practices

• Use the NASA-provided authentication mechanism.
• Use roles or other short-lived credentials.
• Do not place credentials in scripts, repositories, tickets, or documentation.
• Transfer into a designated landing prefix rather than the bucket root.
• Preview directory synchronization operations before running them.
• Do not use destructive synchronization options unless separately reviewed.
• Save transfer logs.
• Validate counts, paths, sizes, and checksums after upload.
• Do not include prohibited or unnecessarily sensitive information in object names.

The following shell example previews an Amazon S3 synchronization, performs the upload, and displays a basic destination summary.

# Use the profile established through the approved authentication process.
# Never insert a permanent access key or secret key into this script.
export AWS_PROFILE="<approved-profile>"

# Preview the proposed upload without changing the destination.
aws s3 sync "/source/dataset" \
  "s3://<approved-bucket>/landing/<dataset-name>/" \
  --dryrun \
  --profile "${AWS_PROFILE}"

# Upload the dataset after reviewing the dry-run result.
aws s3 sync "/source/dataset" \
  "s3://<approved-bucket>/landing/<dataset-name>/" \
  --only-show-errors \
  --profile "${AWS_PROFILE}"

# Display the destination object count and total size.
aws s3 ls \
  "s3://<approved-bucket>/landing/<dataset-name>/" \
  --recursive \
  --summarize \
  --profile "${AWS_PROFILE}"

For important datasets, supplement the summary with a source manifest and checksum validation.

5.2 AWS DataSync

AWS DataSync is a managed transfer service for moving data between supported source systems and AWS storage services. Depending on the current service capabilities, supported sources can include NFS, SMB, HDFS, and compatible object storage. Destinations can include Amazon S3, Amazon Elastic File System, and supported Amazon FSx services.[4]

Recommended for

• Large migrations
• Large directory trees
• NFS or SMB file servers
• Sources containing millions of files
• Scheduled synchronization
• Managed retries and monitoring
• Initial migration followed by incremental updates

Typical process

1. Confirm the dataset’s eligibility and FISMA impact level.
2. Confirm that the destination is appropriate for the data.
3. Deploy or obtain access to a DataSync agent.
4. Configure the source location.
5. Configure the Science Cloud landing location.
6. Configure filters, logging, verification, and bandwidth limits.
7. Run a representative pilot.
8. Review failed, skipped, or changed files.
9. Run the full transfer.
10. Run an incremental transfer if the source remains active.
11. Validate the destination.
12. Promote the data to its authoritative location.

Review current DataSync requirements and supported source and destination combinations before implementation.[4][5]

5.3 AWS Transfer Family

AWS Transfer Family provides managed file-transfer endpoints for supported protocols such as SFTP, FTPS, FTP, and AS2. Received files can be stored in Amazon S3 or Amazon EFS.[6]

Recommended for

• External contributors that require SFTP
• Existing workflows that cannot use an object-storage API
• Recurring file deliveries
• Partner or instrument delivery workflows

Best practices

• Prefer SFTP, FTPS, or AS2.
• Do not use unencrypted FTP.
• Assign a separate identity to each contributor or automated source.
• Restrict each identity to its designated landing path.
• Do not grant contributors access to validated or authoritative data unless required.
• Configure service logging before accepting production transfers.
• Validate received content before downstream processing.
• Ensure contributors understand the Science Cloud’s FISMA limitations and prohibited-data rules.
• Disable accounts and credentials when they are no longer needed.

5.4 Amazon S3 Transfer Acceleration

Amazon S3 Transfer Acceleration can improve some long-distance uploads by using AWS edge locations and the AWS global network.[7]

Recommended for

• Geographically distant upload sources
• Large files transferred over the internet
• Distributed scientific collaborators
• Workloads where latency limits standard S3 upload performance

Considerations

• Test standard S3 transfer performance first.
• Use the AWS speed-comparison tool before enabling the feature.
• Confirm that the transfer client supports the accelerated endpoint.
• Review additional service charges.
• Do not assume that acceleration will improve every transfer.
• Continue to apply the same authentication, encryption, logging, and validation requirements.

Transfer Acceleration is a performance feature. It does not change the destination’s FISMA designation or authorize prohibited data.

5.5 AWS Storage Gateway

Amazon S3 File Gateway presents an NFS or SMB file interface while storing files as objects in Amazon S3.[8]

Recommended for

• Ongoing hybrid file workflows
• Existing applications that require NFS or SMB
• Applications that cannot immediately use Amazon S3 APIs
• Workflows that benefit from local caching

Considerations

• Storage Gateway is generally intended for ongoing workflows rather than a simple one-time upload.
• Test file locking, metadata, permissions, timestamps, and application behavior.
• Size the local cache appropriately.
• Validate how file metadata maps to Amazon S3 objects.
• Confirm that downstream users understand the resulting object-key layout.

For a finite migration, AWS DataSync may be a more appropriate option.

5.6 Offline Transfer

An offline-transfer method may be appropriate when the dataset cannot meet its transfer deadline using available bandwidth.

AWS offline-transfer service availability, eligibility, and regional support can change. Contact the Science Cloud team and consult current AWS documentation before selecting this method.[9]

Required controls

• Confirm data eligibility before copying data to a transfer device.
• Confirm that the destination supports the required FISMA impact level.
• Obtain approval for the physical-transfer process.
• Use approved encryption.
• Establish documented chain-of-custody procedures.
• Secure the device during receipt, use, storage, and return.
• Preserve the authoritative source copy.
• Validate the imported data before accepting it.
• Follow approved procedures for device return or sanitization.

Physical transfer must not be used to bypass FISMA or prohibited-data restrictions.

5.7 Database Transfers

Do not assume that copying database files is sufficient to migrate a database safely.

For supported database engines, consider AWS Database Migration Service or an engine-specific backup and restore process. These approaches can help preserve database consistency and may support ongoing replication during migration.[10]

Before migrating a database:

• Confirm that its records do not contain PII, ITAR-controlled data, or EAR-controlled data.
• Confirm its FISMA impact level.
• Identify dependencies, users, schemas, extensions, and version requirements.
• Test application compatibility.
• Define a consistency-validation and cutover plan.
• Preserve the source until the migrated database is accepted.

6. Recommended Inbound Transfer Pattern

Use the following general pattern:

External source → selected transfer method → landing area → validation → authoritative location → downstream processing

Suggested storage stages

A quarantine area is not an approved location for PII, ITAR-controlled data, or EAR-controlled data. Identify prohibited data before upload.

Where practical, prevent upload identities from modifying validated or authoritative data. Use separate permissions for transfer, validation, administration, and consumption.

7. Prepare the Dataset

7.1 Create an inventory

Record the following before transfer:

• Dataset name and owner
• Source system and source path
• Destination account and storage location
• FISMA impact level
• Confirmation that the data contains no PII
• Confirmation that the data contains no ITAR-controlled information
• Confirmation that the data contains no EAR-controlled information
• Total logical size
• Number of files
• Largest file
• Average file size
• Directory depth
• File formats
• Compression status
• Expected growth rate
• Transfer deadline
• Required retention period
• Whether the source will change during transfer
• Metadata that must be preserved
• Dataset-specific acceptance criteria

File count can be as important as total size. A dataset containing millions of small files may transfer more slowly and generate more storage requests than a similarly sized dataset containing fewer large files.

7.2 Clean the source

Before transfer:

• Remove temporary files that do not need to be retained.
• Remove duplicate or obsolete copies where authorized.
• Resolve files with unsupported names or paths.
• Identify inaccessible or corrupt files.
• Remove prohibited data.
• Confirm that symbolic links and special file types will be handled correctly.
• Document intentionally excluded content.

7.3 Create a manifest

For significant scientific datasets, create a manifest containing:

Protect the manifest from unauthorized modification.

8. Security Best Practices

8.1 Identity and access

• Use NASA-provided federated identities and IAM roles.
• Use short-lived credentials.
• Apply least-privilege access.
• Restrict upload identities to the required bucket, prefix, or file system.
• Separate upload, validation, administration, and consumption permissions.
• Require multifactor authentication where applicable.
• Do not place credentials in scripts, command-line arguments, tickets, repositories, or shared documents.
• Remove temporary permissions after the migration.
• Do not make storage public merely because the dataset is FISMA Low.
• Use Amazon S3 Block Public Access where appropriate.[11]

8.2 Encryption

• Use encrypted service endpoints.
• Use the destination account’s established encryption configuration.
• Use an approved AWS Key Management Service key where required.
• Ensure that transfer identities receive only the necessary key permissions.
• Do not place prohibited or unnecessarily sensitive information in filenames, object keys, tags, or logs.

Amazon S3 encrypts new object uploads at rest by default. Projects must still use the destination account’s established key-management and access-control configuration.[12]

8.3 Logging and monitoring

Use the logging capabilities required by the destination environment, which may include:

• AWS CloudTrail
• CloudTrail Amazon S3 data events
• AWS DataSync task logs
• AWS Transfer Family logs
• Amazon CloudWatch metrics and alarms
• Amazon S3 Inventory
• Account-specific security monitoring

CloudTrail S3 data events can generate substantial log volume and cost. Scope them according to the destination environment’s requirements.[13]

9. Data Integrity and Validation

A successful transfer command or task status does not prove that the dataset is complete, correct, or scientifically usable.

Minimum validation

Compare the source and destination using:

• File or object count
• Total logical size
• Relative paths or object keys
• Expected filenames
• Unexpected zero-byte files
• Failed or skipped files
• Transfer warnings and retry logs
• A representative content sample

Checksum validation

Use checksums when end-to-end integrity verification is required. Amazon S3 supports checksum calculation and validation, but behavior varies by operation and client. Confirm that the selected transfer method uses the required checksum algorithm and validates the complete object.[14]

For multipart uploads, do not assume that an Amazon S3 ETag is always an MD5 checksum of the complete source file.

Scientific validation

Where applicable, also verify:

• File format readability
• Metadata completeness
• Record counts
• Variables and units
• Coordinate reference systems
• Spatial coverage
• Temporal coverage
• Calibration information
• Processing level
• Dataset version
• Provenance
• Expected relationships between files

Do not delete or alter the source copy until:

1. Integrity checks have passed.
2. Scientific validation has passed.
3. Access controls have been reviewed.
4. The data owner has accepted the cloud copy.
5. Source deletion has been separately authorized.

10. Performance Best Practices

• Run a representative pilot before the full transfer.
• Use multipart upload for large Amazon S3 objects.
• Use controlled parallelism rather than a single serial copy process.
• Avoid excessive parallelism that overwhelms the source system.
• Place transfer agents close to source storage where applicable.
• Schedule large transfers outside peak source-system usage periods.
• Monitor source disk, processor, memory, and bandwidth utilization.
• Use service bandwidth controls when necessary.
• Test S3 Transfer Acceleration before paying for production use.
• Avoid repeatedly rescanning large directory trees with multiple tools.
• Investigate failed files rather than repeatedly retransmitting the entire dataset.
• Retain transfer logs and task identifiers.

Packaging many small files into larger containers can improve throughput, but do so only when compatible with:

• Downstream access patterns
• Metadata requirements
• Retention policies
• Incremental updates
• Partial retrieval
• Scientific processing tools

Many scientific file formats are already compressed. Test compression before adding it to the transfer workflow.

11. Cost Best Practices

Potential charges include:

• Transfer-service usage
• Amazon S3 storage
• Amazon S3 API requests
• AWS Key Management Service requests
• S3 Transfer Acceleration
• DataSync usage
• Transfer Family endpoints and transfers
• Offline-transfer services
• CloudTrail data events
• Cross-Region data movement
• Archival retrieval or early-deletion charges
• Failed multipart uploads that remain incomplete

Before starting a large transfer:

• Estimate the expected storage and service cost.
• Include file count when estimating request charges.
• Apply project tags to temporary transfer resources.
• Configure budgets or cost alerts where available.
• Avoid accidental cross-Region copies.
• Remove unused endpoints, agents, and temporary resources.
• Configure an approved policy for aborting incomplete multipart uploads.
• Do not place data into an archival storage class until validation and access requirements are understood.

Review the current AWS pricing pages because costs vary by service, Region, and usage pattern.[15]

12. Microsoft Azure Options

The same NASA Science Cloud restrictions apply to approved Azure environments:

• FISMA Low is supported by default.
• FISMA Moderate is supported only in specifically designated environments.
• PII is not permitted.
• ITAR-controlled data is not permitted.
• EAR-controlled data is not permitted.
• Azure service certifications do not expand the authorization of a NASA environment.

Azure service selection

12.1 AzCopy

AzCopy is a command-line utility for copying data to and from Azure Blob Storage and Azure Files.[16]

Recommended practices include:

• Use Microsoft Entra ID authentication where supported.
• Prefer short-lived authentication.
• Do not embed storage account keys or shared access tokens in scripts.
• Use a designated landing container or path.
• Capture transfer logs.
• Validate file counts, sizes, paths, and checksums.
• Review synchronization options before using them against existing destination data.

12.2 Azure Storage Mover

Azure Storage Mover provides managed migration workflows for supported on-premises file sources and Azure Storage destinations.[17]

It may be appropriate for:

• Large directory trees
• NFS or SMB migration, subject to current service support
• Repeatable migration jobs
• Managed transfer monitoring
• Initial migration followed by additional transfer runs

Verify current source, destination, metadata, and regional support before selecting the service.

12.3 Azure Data Box

Azure Data Box provides physical devices for transferring large datasets into Azure.[18]

Use of a physical device requires:

• Data eligibility confirmation
• Approval of the physical-transfer process
• Approved encryption
• Documented chain of custody
• Secure handling and storage
• Post-import validation
• Retention of the source until acceptance
• Approved return or sanitization procedures

12.4 Azure Data Factory

Azure Data Factory can copy data between supported data stores and orchestrate recurring ingestion workflows.[19]

It may be appropriate for:

• Structured and semi-structured data
• Scheduled ingestion
• Transfers between heterogeneous data stores
• Data movement combined with transformation or orchestration

Confirm that the complete workflow and every intermediate location are appropriate for the data’s FISMA impact level.

13. Inbound Transfer Checklist

Before transfer

• The data owner has identified the applicable FISMA impact level.
• The dataset does not contain PII.
• The dataset does not contain ITAR-controlled data.
• The dataset does not contain EAR-controlled data.
• FISMA Low data is going to an appropriate Science Cloud account.
• FISMA Moderate data is going only to a specifically designated Moderate account.
• The destination storage location and landing path are known.
• The transfer method has been selected.
• The dataset’s total size and file count are known.
• The largest and average file sizes are known.
• Authentication and access controls are configured.
• Encryption is configured.
• Logging and monitoring are configured.
• A source manifest has been created where appropriate.
• Acceptance criteria have been documented.
• A representative pilot has succeeded.

During transfer

• Transfer throughput and status are monitored.
• Source-system health is monitored.
• Failed, skipped, and changed files are recorded.
• Logs and task identifiers are retained.
• Unexpected content causes the transfer to be paused and investigated.
• Destination storage growth and transfer costs are monitored.
• Configuration changes are documented.

After transfer

• Source and destination counts have been compared.
• Source and destination logical sizes have been compared.
• Paths or object keys have been validated.
• Checksums or manifest records have been validated.
• Transfer errors and warnings have been resolved.
• Scientific validation has been completed.
• Destination permissions have been reviewed.
• Encryption and logging have been confirmed.
• The data owner has accepted the cloud copy.
• Data has been moved or promoted to its authoritative location.
• Temporary permissions and resources have been removed.
• The source remains available until deletion is formally authorized.

14. References

[1] National Institute of Standards and Technology, “FIPS PUB 199: Standards for Security Categorization of Federal Information and Information Systems,” https://csrc.nist.gov/pubs/fips/199/final

[2] AWS, “Using high-level S3 commands with the AWS CLI,” https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html

[3] AWS, “Uploading and copying objects using multipart upload,” https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

[4] AWS, “What is AWS DataSync?” https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html

[5] AWS, “AWS DataSync requirements,” https://docs.aws.amazon.com/datasync/latest/userguide/datasync-requirements.html

[6] AWS, “What is AWS Transfer Family?” https://docs.aws.amazon.com/transfer/latest/userguide/what-is-aws-transfer-family.html

[7] AWS, “Configuring fast, secure file transfers using Amazon S3 Transfer Acceleration,” https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

[8] AWS, “What is Amazon S3 File Gateway?” https://docs.aws.amazon.com/filegateway/latest/files3/what-is-file-s3.html

[9] AWS, “Cloud data migration services,” https://aws.amazon.com/cloud-data-migration/

[10] AWS, “What is AWS Database Migration Service?” https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html

[11] AWS, “Blocking public access to Amazon S3 storage,” https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html

[12] AWS, “Protecting data with server-side encryption,” https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html

[13] AWS, “Logging Amazon S3 API calls using AWS CloudTrail,” https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudtrail-logging.html

[14] AWS, “Checking object integrity in Amazon S3,” https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

[15] AWS, “AWS data transfer charges,” https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer

[16] Microsoft, “Get started with AzCopy,” https://learn.microsoft.com/azure/storage/common/storage-use-azcopy-v10

[17] Microsoft, “What is Azure Storage Mover?” https://learn.microsoft.com/azure/storage-mover/service-overview

[18] Microsoft, “What is Azure Data Box?” https://learn.microsoft.com/azure/databox/data-box-overview

[19] Microsoft, “Introduction to Azure Data Factory,” https://learn.microsoft.com/azure/data-factory/introduction