DynamoDB

Availability: Core Standard Plus Pro Enterprise Flex PyAirbyte
Connector type (beta): Data replication Agent(learn more)
Support Level: Airbyte
Connector Version: 0.4.1 (last updated 9 days ago)
Sync Success Rate: High
Usage Rate: High
Data workers used: 0.5 per sync (Learn more)
Definition ID: 50401137-8871-4c5a-abb7-1f5fda35545a

The DynamoDB source connector reads the tables of an Amazon DynamoDB account and region. It supports full refresh and incremental syncs, infers each table's schema by sampling its items, reads a table page by page with the Scan API, and saves its position after every round of pages so that a large table resumes after an interruption instead of starting over.

Version 0.4.0 rebuilds the connector on Airbyte's Bulk CDK. It accepts the saved state of versions 0.3.x and any configuration that uses an access key and a region, so those connections keep syncing without a reset. A source that used the role-based authentication of versions 0.3.x, or that has no region, fails until you edit it. See Changes in 0.4.0 for what changed and what to do.

Features

Feature	Supported	Notes
Full refresh sync	Yes	Resumable, see State and resuming
Incremental sync - append	Yes	Cursor based, see Incremental sync
Incremental sync - append + deduped	Yes	Removes duplicates by the table's partition key, see Limitations for composite keys
Replicate deletes	No	The connector doesn't read DynamoDB Streams
Change data capture (CDC)	No
Namespaces	No	A region has a flat list of tables
Primary keys	Yes	The table's partition key
Checkpoints	Yes	A state message after every round of scan pages, per scan segment
Speed mode	Yes	The connector supports the socket data channel with JSONL and protocol buffer serialization

The connector is read-only. It calls ListTables, DescribeTable, and Scan, and never writes to your tables.

Getting started

Requirements

An AWS account with the DynamoDB tables to sync.
An IAM identity with the permissions listed below, and its access key. Credentials always come from the source configuration: the connector can't use an AWS profile, an instance role, or environment variables of the machine it runs on.

IAM permissions

Create a dedicated IAM user or role for Airbyte and grant it the following permissions. Restrict DescribeTable and Scan to the tables you sync when you don't want Airbyte to see the others.

Permission	Resource	Why
`dynamodb:ListTables`	`*`	Lists the tables of the region during the connection test and discovery
`dynamodb:DescribeTable`	Each table to sync	Reads the table's key schema, which becomes the stream's primary key
`dynamodb:Scan`	Each table to sync	Samples items during discovery and reads the data during a sync
`sts:AssumeRole`	The role, with Access Key and IAM Role	Lets the access key's identity assume the role that reads the tables

For example, this identity-based policy grants read access to two tables. Replace the region, account ID, and table names with your own.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "dynamodb:ListTables",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:DescribeTable", "dynamodb:Scan"],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:123456789012:table/orders",
        "arn:aws:dynamodb:us-east-1:123456789012:table/customers"
      ]
    }
  ]
}

With Access Key and IAM Role, attach this policy to the role, and give the IAM user that owns the access key permission to call sts:AssumeRole on that role.

Authentication

The connector offers two authentication methods. Both start from an access key.

Access Key: the access key ID and secret access key of an IAM user, or temporary credentials issued by AWS STS. For temporary credentials, also fill in the Session Token. The connector signs every request with this key. To create an access key, see Manage access keys for IAM users.
Access Key and IAM Role: an access key that is allowed to call sts:AssumeRole on an IAM role, plus the role's ARN and, when the role's trust policy requires one, the external ID. The connector assumes the role and reads the tables with the role's temporary credentials. Use this method for cross-account access or to keep the reading identity to a least-privilege role. The role's trust policy must allow the access key's identity to assume it. See Grant a user permissions to switch roles.

The role-based authentication of versions 0.3.x, which took credentials from the environment the connector runs in, isn't available. A saved configuration that uses it fails the connection test until you edit it to use one of the two methods. See What to do after upgrading.

Set up the DynamoDB source in Airbyte

In the Airbyte UI, click Sources and then + New source.
Select DynamoDB from the source type list.
Fill in the connection fields:

Field	Required	Description
Credentials	Yes	Access Key or Access Key and IAM Role, see Authentication.
AWS Region	Yes	The region of the tables, for example `us-east-1`. The region is required even with an endpoint override.
DynamoDB Endpoint	No	An `http` or `https` URL that replaces the public endpoint of the region: a VPC endpoint, a FIPS endpoint, or a DynamoDB Local instance.
Reserved attribute names	No	Kept so that configurations of versions 0.3.x still load. The connector ignores it: every attribute is aliased in its scan expressions, so reserved words and special characters need no configuration.
Ignore missing read permissions tables	No	When on, discovery skips a table whose `Scan` is denied instead of failing. Off by default.
Discovery sample size (Advanced)	No	How many items discovery reads from each table to infer its attributes and their types. Defaults to 1000, the range is 1 to 100000. A larger sample finds rarer attributes and consumes more read capacity.
Checkpoint Target Time Interval (Advanced)	No	How often, in seconds, a stream saves its position. A table is scanned in rounds of about this duration and a state message follows every round. Defaults to 300, the minimum is 1.
Max Concurrent Queries to Database	No	Airbyte Cloud only. How many `Scan` requests run at the same time, across the segments of a large table and across tables. Leave empty to let Airbyte choose. Self-managed Airbyte shows the field disabled and scans one table at a time. See Concurrency.

Click Set up source. The connection test lists the tables of the region. It fails with Discovered zero tables. when the identity can't see any table there.

Replication

Discovered streams

Every table of the region is a stream. DynamoDB tables have no schema, only their key attributes are typed, so the connector infers a schema the way versions 0.3.x did: it scans up to Discovery sample size items of the table, merges the top-level attributes of every sampled item, and maps each attribute to a JSON schema from the type of its value. When an attribute has values of different types in different items, the last sampled item decides its type. Nested maps and lists keep the structure of the sampled values. An attribute that appears only in items outside the sample is missing from the schema. A table without items is discovered with its key attributes only, typed from DescribeTable; its other attributes appear at the first schema refresh after it has items.

The stream's primary key is the table's partition key. A table with a composite key reports only its partition key, as versions 0.3.x did, so more than one item can share a primary key value. Global and local secondary indexes are never read.

Full refresh

A full refresh scans the whole table with the Scan API, one page of up to 1 MB at a time, projecting exactly the attributes of the stream's schema. Every attribute is referenced through a placeholder in the expression, so attribute names that are DynamoDB reserved words, or that contain spaces, dots, dashes, or a leading underscore, work without configuration.

Scan is eventually consistent and has no snapshot isolation. An item written while its table is being scanned may appear in either version, and an item inserted or deleted during the scan may or may not be part of the result.

Incremental sync

Incremental sync uses a cursor attribute that you choose for each stream. The cursor must be a top-level attribute of type string (S), for example an ISO 8601 timestamp, or number (N), for example an epoch value. The connector takes the cursor's type from the stream's schema: a string attribute is compared as a string and an integer or number attribute as a number with full DynamoDB precision. Nested attributes and other types can't be cursors.

The first sync reads the whole table. Every later sync scans the whole table again with a filter that keeps the items whose cursor is greater than the saved cursor value, or greater than or equal to it when the saved value is a plain date such as 2016-02-15, so that the items of the day of the last sync aren't skipped. DynamoDB applies the filter after reading the items, so an incremental sync consumes as much read capacity as a full refresh and takes about as long; the filter only reduces the records sent to the destination. The highest cursor value read becomes the saved cursor once the scan is complete. Items without the cursor attribute are replicated but don't move the cursor, and changing the cursor attribute of a stream starts it over.

Choose an attribute whose values only grow and that every new or updated item carries, such as an update timestamp. Because the cursor is compared as text for string attributes, use a fixed-width format that sorts chronologically, such as 2024-05-01T10:00:00Z. Cursor values should never be empty.

State and resuming

The connector emits state while it reads, not only at the end of a sync. After every round of scan pages, about Checkpoint Target Time Interval long, the state records, for every segment of the table, the key of the last page read or that the segment is complete; for an incremental stream it also carries the highest cursor value seen so far in each segment. A retried attempt resumes every unfinished segment at its saved key, so at most one page per segment is read twice and no item is skipped. The number of segments is saved with the state and kept until the table is complete, whatever the Max Concurrent Queries to Database setting of the retry. A completed incremental stream saves its cursor value in the same shape versions 0.3.x used, and a completed full refresh saves a marker so that a retried attempt doesn't read the table again. Resetting the connection's data makes the next sync start from the beginning.

The connector also reads the state saved by versions 0.3.x. A stream with such state resumes after its saved cursor value instead of reading the table from the start.

Concurrency

Parallel scans and concurrent scans are an Airbyte Cloud feature. Self-managed Airbyte reads every table with one sequential scan and one table at a time, as versions 0.3.x did; the Max Concurrent Queries to Database field is shown disabled there and a value set through the API is ignored.

On Airbyte Cloud the connector reads a large table with a parallel scan, where DynamoDB divides the table into segments by hashing each item's partition key and the connector scans each segment on its own. The segments cost nothing to compute and consume the same read capacity as one sequential scan. The number of segments comes from the table size that DescribeTable reports, one segment per 64 MB, up to 128 segments per table, so a table under 64 MB is still read by a single scan. DynamoDB refreshes that size about every six hours, so a table loaded since the last refresh reports zero bytes; the connector then uses as many segments as Max Concurrent Queries to Database.

Max Concurrent Queries to Database sets how many scans run at the same time across all tables. Left empty, Airbyte reads one table at a time on the standard data channel and one table per socket in speed mode. At 1 the segments of a table are read one after another and tables one after another, which is as fast as a single sequential scan. Raise it to read segments and tables in parallel; the speed-up is close to linear until the table's read capacity or the network is saturated. Each running scan holds one page of up to 1 MB in memory.

Segments can be uneven. All items with the same partition key land in the same segment, so a table with few partition keys, or a few large item collections, has segments of different sizes; the larger ones just finish later.

Read throughput

Measured on 2026-09-24 against a 1 GB table in us-east-2 (1,000,000 items of about 1 KB) from a laptop on a residential connection, where one 1 MB scan page takes about 0.4 s to arrive; inside AWS the absolute numbers are higher, the ratios are what matter. Every run read the same 1,000,000 records and consumed the same 124,274 read units (about $0.03 on-demand). Concurrent scans are an Airbyte Cloud feature; self-managed Airbyte reads at the rate of the first row.

Concurrent scans	Segments read in parallel	Wall time	Throughput
1	1	7 min 47 s	2.1 MB/s
4	4	1 min 57 s	8.5 MB/s
8	8	1 min 12 s	14 MB/s
16	16	45 s	23 MB/s

A sync interrupted at concurrency 8 and resumed from its last state returned every one of the 1,000,000 keys exactly once, apart from the records read in the 10 seconds after the last checkpoint. Sixteen concurrent segments completed inside a 384 MB Java heap.

Read capacity

Every sync scans each configured table in full, and so does discovery for up to Discovery sample size items per table. Scan consumes read capacity for every item it reads, whatever the projection or filter: an eventually consistent read of up to 4 KB costs half a read request unit, see Read/write capacity mode. On a table with provisioned capacity a sync competes with your own workload for read capacity units; DynamoDB throttles the connector when the table's capacity is exceeded, and the connector retries and, when the throttling persists, reports a transient error that Airbyte retries. Raise the table's read capacity or switch it to on-demand mode when syncs are throttled.

Data type mapping

Each top-level attribute of a table becomes a field whose JSON schema and value follow the type of the attribute's value in the sampled items. The schemas use the same shapes as Airbyte's other database connectors: {"type": "string"} rather than a ["null", "string"] type list, and whole numbers as {"type": "number", "airbyte_type": "integer"}. Every field can hold null, as in all Airbyte sources.

DynamoDB type	Airbyte type	Notes
`S` (string)	`string`
`N` (number)	`integer`	When the sampled value is an integer that fits in 64 bits. Values are read as integers when they fit in 64 bits, otherwise as exact decimals with the full 38 digits of precision that DynamoDB stores.
`N` (number)	`number`	When the sampled value has a fraction or an exponent, or doesn't fit in 64 bits.
`B` (binary)	`string`	Base64 encoded, with `contentEncoding: base64` in the schema.
`BOOL`	`boolean`
`NULL`	`null`
`M` (map)	`object`	The schema lists the nested attributes of the sampled value with their own types.
`L` (list)	`array`	The schema lists one item schema per element of the sampled value.
`SS` (string set)	`array`	Items are strings.
`NS` (number set)	`array`	Items are numbers, read exactly.
`BS` (binary set)	`array`	Items are base64 encoded strings.

An attribute that a record's item doesn't have is null in the record.

Changes in 0.4.0

Version 0.4.0 loads the access key configurations and the state of versions 0.3.x, and its discover output lists the same streams, fields, and types for the same tables, with these differences. What to do after upgrading lists the few cases that need an action.

Schemas use Airbyte's standard shapes. Versions 0.3.x wrote {"type": ["null", "string"]} and {"type": ["null", "integer"]}; this connector writes {"type": "string"} and {"type": "number", "airbyte_type": "integer"}, like every other Airbyte database connector. A connection upgraded from 0.3.x keeps syncing with its saved catalog and shows a schema change on every column at its next schema refresh; the column types in the destination don't change.
Credentials come from the configuration only. Role-based authentication, which used the credentials of the environment the connector ran in, is gone: a source configured without an access key fails the connection test with a message that says so. Temporary credentials with a session token and role assumption with sts:AssumeRole are new, AWS Region is required and DynamoDB Endpoint must be an http or https URL, which versions 0.3.x required in practice too.
Reserved attribute names need no configuration. Every attribute is aliased in the scan expressions, so the Reserved attribute names field is ignored. Versions 0.3.x failed on any unlisted reserved word or special character and could never read a table that had both field.name and field-name.
Large numbers are exact. An N value that doesn't fit in 64 bits is read as an exact decimal; versions 0.3.x converted it to a floating point number and lost precision. Rows that versions 0.3.x already wrote keep their rounded value until the table is read in full again, and equality joins on such values may start or stop matching.
Absent attributes are null. An attribute that an item doesn't have is present in the record as null; versions 0.3.x left it out. Destinations that keep the record as JSON, such as object storage, now show the attribute as an explicit null key.
Integer cursors work. Versions 0.3.x failed before the first record on any cursor attribute discovered as integer, which is every whole number.
Syncs resume. Full refresh and incremental streams save their position after every round of scan pages and resume there; versions 0.3.x saved one state at the end of an incremental stream and nothing for a full refresh.
Failures carry a message. The connection test reports why it failed; versions 0.3.x reported a failure without a message. A configured table that no longer exists, or whose stream has no fields, fails only its own stream; the other streams of the sync are read.
An empty table is discovered with its key attributes only. Its other attributes appear at the first schema refresh after the table has items. Versions 0.3.x listed an empty table as a stream without fields and couldn't read it. The connection test fails when the region has no tables.
Large tables are read with parallel scans on Airbyte Cloud. A table over 64 MB is split into scan segments that Max Concurrent Queries to Database reads in parallel; self-managed Airbyte, like versions 0.3.x, reads every table with one sequential scan.
Discovery sample size and Max Concurrent Queries to Database are new settings; versions 0.3.x always sampled 1000 items and read one table at a time.

What to do after upgrading

A source that uses an access key and has a region set needs no action: its connections keep syncing with their saved catalog, and incremental streams continue from their saved cursor value.

If a source used role-based authentication, edit it: choose Access Key and enter the access key of an IAM user or temporary credentials with a session token, or choose Access Key and IAM Role and enter an access key that may assume the role the connector should read with. See IAM permissions for what the identity needs. Until then the connection test and every sync of that source fail with a message that says so.
If AWS Region is empty, set it to the region of the tables.
Refreshing the source schema of a connection is recommended but optional. It shows a schema change on every column, picks up the key attributes of tables that were empty, and leaves the column types in the destination as they are.
Clearing a stream's data is optional and only worthwhile when its table has N values wider than 64 bits that versions 0.3.x wrote as rounded numbers. The next sync reads the whole table again.

Limitations and troubleshooting

No change data capture. Deleted items aren't detected and updated items are only picked up when the cursor attribute changes. The connector doesn't read DynamoDB Streams.
Every sync scans every configured table in full, incremental syncs included, and consumes read capacity for every item of the table. See Read capacity.
The schema comes from a sample. Raise Discovery sample size and refresh the source schema when an attribute is missing from a stream or has the wrong type, and pin the type of an attribute that varies between items by storing one type only.
Composite keys report only the partition key as the primary key. Destinations that remove duplicates by primary key collapse the items that share a partition key value; use append modes for such tables.
The access key id or the session token is invalid or The secret access key is invalid mean the configured access key is wrong; The temporary credentials (session token) have expired means new temporary credentials are needed.
... is not authorized to perform: sts:AssumeRole ... names the identity and the role: the role's trust policy doesn't allow the access key's identity to assume it, or the external ID is wrong. ... is not authorized to perform: dynamodb:... names a missing permission from the IAM permissions table.
Could not reach the DynamoDB endpoint means the configured endpoint doesn't resolve or accept connections; check DynamoDB Endpoint and AWS Region.
A configured table does not exist (any more) appears when a table was deleted or renamed after the source schema was set up; refresh the source schema. The saved state of a stream does not match its table's key schema appears when a table was recreated with other key attributes, and ... does not match its table's scan segments when a saved segment key no longer fits the table; reset the stream's data in both cases.
Throttling errors (ProvisionedThroughputExceededException, ThrottlingException) are transient. Airbyte retries the sync; raise the table's read capacity if they persist.

IP allow list

If you use Airbyte Cloud and your organization restricts access to specific IPs, add the Airbyte Cloud IP addresses to your allow list.

Reference

Config fields reference

Field

Type

Property name

object

credentials

string

region

integer

checkpoint_target_interval_seconds

integer

discover_sample_size

string

endpoint

boolean

ignore_missing_read_permissions_tables

integer

max_db_connections

string

reserved_attribute_names

Changelog

Expand to review

Version	Date	Pull Request	Subject
0.4.1	2026-10-07	88146	Promote the connector to generally available and certified
0.4.0	2026-10-06	86997	Rebuild on the Bulk CDK: parallel scan segments on Airbyte Cloud, resumable full refresh and incremental syncs, speed mode, exact numbers, integer cursors, temporary credentials and IAM role assumption. Role-based authentication is removed, see Changes in 0.4.0
0.3.11	2025-07-11	62916	Add gradle docker plugins
0.3.10	2025-06-17	61601	fix(source-dynamodb): Replace ListNode with Iterator for lazyness #61600
0.3.9	2025-02-12	53202	fixed IRSA by adding STS to classpath of connector.
0.3.8	2025-01-13	51489	Use a non root base image
0.3.7	2024-12-19	49881	Use a base image: airbyte/java-connector-base:1.0.0
0.3.6	2024-07-31	41936	Fix incorrect type check for incremental read
0.3.5	2024-07-23	42433	add PR number
0.3.4	2024-07-23	42433	fix primary key fetching
0.3.3	2024-07-22	42398	fix primary key fetching
0.3.2	2024-05-01	27045	Fix missing scan permissions
0.3.1	2024-05-01	31935	Fix list more than 100 tables
0.3.0	2024-04-30	37530	Allow role based access
0.2.3	2024-02-14	35232	Adopt CDK 0.20.4
0.2.2	2024-02-01	34453	bump CDK version
0.2.1	2024-01-05	33924	Add new ap-southeast-3 AWS region
0.2.0	2023-12-19	33485	Remove LEGACY state
0.1.2	2023-02-27	20172	Fix reserved words in projection expression & make them configurable
0.1.1	2023-02-10	22682	Fix build
0.1.0	2022-11-14	18750	Initial version

Features​

Getting started​

Requirements​

IAM permissions​

Authentication​

Set up the DynamoDB source in Airbyte​

Replication​

Discovered streams​

Full refresh​

Incremental sync​

State and resuming​

Concurrency​

Read throughput​

Read capacity​

Data type mapping​

Changes in 0.4.0​

What to do after upgrading​

Limitations and troubleshooting​

IP allow list​

Reference​

Config fields reference

Changelog​

Features

Getting started

Requirements

IAM permissions

Authentication

Set up the DynamoDB source in Airbyte

Replication

Discovered streams

Full refresh

Incremental sync

State and resuming

Concurrency

Read throughput

Read capacity

Data type mapping

Changes in 0.4.0

What to do after upgrading

Limitations and troubleshooting

IP allow list

Reference

Changelog