Transfer dataΒΆ

Here, we’ll show how to transfer data from another instance into the current instance.

!lamin init --storage ./test-transfer --schema bionty
Hide code cell output
πŸ’‘ connected lamindb: anonymous/test-transfer
import lamindb as ln

ln.settings.transform.stem_uid = "ITeOtm7bhtdq"
ln.settings.transform.version = "1"
ln.track()
Hide code cell output
πŸ’‘ connected lamindb: anonymous/test-transfer
πŸ’‘ notebook imports: lamindb==0.72.1
πŸ’‘ saved: Transform(uid='ITeOtm7bhtdq5zKv', version='1', name='Transfer data', key='transfer', type='notebook', created_by_id=1, updated_at='2024-05-25 15:25:32 UTC')
πŸ’‘ saved: Run(uid='UgJjwUTClVtI1VvlqyMt', transform_id=1, created_by_id=1)
Run(uid='UgJjwUTClVtI1VvlqyMt', started_at='2024-05-25 15:25:32 UTC', is_consecutive=True, transform_id=1, created_by_id=1)

All artifacts in the laminlabs/cellxgene clone of CZ CELLxGENE (for more info, see cellxgene):

artifacts = ln.Artifact.using("laminlabs/cellxgene")
artifacts.df().head()
Hide code cell output
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1717 6GM0siRkIUISUwpFb54A 2023-07-25 Pla_HDBR9518710 cell-census/2023-07-25/h5ads/ff12e239-9292-4d2... .h5ad AnnData 21993955 PFgOHr86dGrEOrPELpKvrQ-3 md5-n None 1216.0 1 False 2 11.0 16.0 1 2024-01-24 07:05:41.771446+00:00
1613 qnlBU5a6OR46Zkku1gEl 2023-07-25 WS_PLA_S9101764 cell-census/2023-07-25/h5ads/dfe8e072-94a7-415... .h5ad AnnData 39725919 -2OE3CsXIxegNGOQO7cJVw-5 md5-n None 3568.0 1 False 2 11.0 16.0 1 2024-01-24 07:05:43.770581+00:00
1497 w6UNsyZp0eTrl3Uz15Wl 2023-07-25 WS_PLA_S9101769 cell-census/2023-07-25/h5ads/c1568274-2af2-4cd... .h5ad AnnData 34328778 oaxWjROHsHneVVf7VgcoaA-5 md5-n None 3130.0 1 False 2 11.0 16.0 1 2024-01-24 07:05:45.247888+00:00
1434 txst3MWWFtvPGcYOR145 2023-07-25 primary_trophoblast_organoid cell-census/2023-07-25/h5ads/ab326369-b63c-48d... .h5ad AnnData 530799215 2i7cuX3r562-mfIK8GYAyA-64 md5-n None 26853.0 1 False 2 11.0 16.0 1 2024-01-24 07:05:45.699596+00:00
979 Vd2uGOTQOwleGdFRlEOp 2023-07-25 Heart - A single-cell transcriptomic atlas cha... cell-census/2023-07-25/h5ads/1fe63353-9e75-482... .h5ad AnnData 148387585 4keMLCOrr6OP70fPvE0TIQ-18 md5-n None 8613.0 1 False 2 11.0 16.0 1 2024-01-24 07:15:09.559452+00:00

Query or search the queryset:

artifact = artifacts.filter(description__icontains="tabula sapiens").first()
artifact
Hide code cell output
Artifact(uid='7JWPBDDaO3v4bsarcj50', version='2023-07-25', description='Tabula Sapiens - Salivary_Gland', key='cell-census/2023-07-25/h5ads/f01bdd17-4902-40f5-86e3-240d66dd2587.h5ad', suffix='.h5ad', accessor='AnnData', size=575121838, hash='4DpycNCrKc_Q_pEYmPXZRw-69', hash_type='md5-n', n_observations=27199, visibility=1, key_is_virtual=False, created_by_id=1, storage_id=2, transform_id=11, run_id=16, updated_at='2024-01-24 07:05:50 UTC')

Save the artifact to the default instance:

artifact.save()
Hide code cell output
πŸ’‘ returning existing Organism(uid='1dpCL6Td') on default database
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
πŸ’‘ returning existing Tissue(uid='704PM5QF') on default database
πŸ’‘ returning existing Tissue(uid='3mCb7TBl') on default database
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
πŸ’‘ returning existing CellType(uid='28V22coI') on default database
πŸ’‘ returning existing CellType(uid='1xjnFaa2') on default database
πŸ’‘ returning existing CellType(uid='4PSMdO3I') on default database
πŸ’‘ returning existing CellType(uid='puGNwNrs') on default database
πŸ’‘ returning existing CellType(uid='6cbuZBUM') on default database
πŸ’‘ returning existing CellType(uid='6Q3j6SOp') on default database
πŸ’‘ returning existing CellType(uid='6ujMwy7s') on default database
πŸ’‘ returning existing CellType(uid='7Crr32HI') on default database
πŸ’‘ returning existing CellType(uid='6dzoXJ3Y') on default database
πŸ’‘ returning existing CellType(uid='6rfrjhvo') on default database
πŸ’‘ returning existing CellType(uid='ryEtgi1y') on default database
πŸ’‘ returning existing CellType(uid='1V5wVqK5') on default database
πŸ’‘ returning existing CellType(uid='22LvKd01') on default database
πŸ’‘ returning existing CellType(uid='zQ4dyjEs') on default database
πŸ’‘ returning existing CellType(uid='2cUPBtY8') on default database
πŸ’‘ returning existing CellType(uid='6IC9NGJE') on default database
πŸ’‘ returning existing CellType(uid='2nPA0h4F') on default database
πŸ’‘ returning existing CellType(uid='7Ql7OzRU') on default database
πŸ’‘ returning existing CellType(uid='3jdCg7zi') on default database
πŸ’‘ returning existing CellType(uid='1J6s4gSi') on default database
πŸ’‘ returning existing CellType(uid='3el7r2gM') on default database
πŸ’‘ returning existing CellType(uid='531hEapj') on default database
πŸ’‘ returning existing CellType(uid='4pk08hOx') on default database
πŸ’‘ returning existing CellType(uid='3muVF5Bn') on default database
❗ did not create Disease record for 1 non-validated ontology_id: 'PATO:0000461'
❗ did not create Phenotype records for 2 non-validated ontology_ids: 'PATO:0000384', 'PATO:0000383'
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
πŸ’‘ returning existing ExperimentalFactor(uid='1cdYMnKh') on default database
πŸ’‘ returning existing ExperimentalFactor(uid='4FDIH9Sv') on default database
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
πŸ’‘ returning existing DevelopmentalStage(uid='2zxrb1Pu') on default database
πŸ’‘ returning existing DevelopmentalStage(uid='5q64GXx1') on default database
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
πŸ’‘ returning existing Ethnicity(uid='15mGvAtW') on default database
Artifact(uid='7JWPBDDaO3v4bsarcj50', version='2023-07-25', description='Tabula Sapiens - Salivary_Gland', key='cell-census/2023-07-25/h5ads/f01bdd17-4902-40f5-86e3-240d66dd2587.h5ad', suffix='.h5ad', accessor='AnnData', size=575121838, hash='4DpycNCrKc_Q_pEYmPXZRw-69', hash_type='md5-n', n_observations=27199, visibility=1, key_is_virtual=False, created_by_id=1, storage_id=2, transform_id=1, run_id=1, updated_at='2024-05-25 15:25:34 UTC')

All features & labels have been transferred, the data itself is still on CZ’s S3:

artifact.describe()
Hide code cell output
Artifact(uid='7JWPBDDaO3v4bsarcj50', version='2023-07-25', description='Tabula Sapiens - Salivary_Gland', key='cell-census/2023-07-25/h5ads/f01bdd17-4902-40f5-86e3-240d66dd2587.h5ad', suffix='.h5ad', accessor='AnnData', size=575121838, hash='4DpycNCrKc_Q_pEYmPXZRw-69', hash_type='md5-n', n_observations=27199, visibility=1, key_is_virtual=False, updated_at='2024-05-25 15:25:34 UTC')
  Provenance
    .created_by = 'anonymous'
    .storage = 's3://cellxgene-data-public'
    .transform = 'Transfer data'
    .run = '2024-05-25 15:25:32 UTC'
  Labels
    .organisms = 'human'
    .tissues = 'sublingual gland', 'parotid gland'
    .cell_types = 'monocyte', 'myoepithelial cell', 'CD4-positive, alpha-beta T cell', 'endothelial cell of lymphatic vessel', 'duct epithelial cell', 'endothelial cell of hepatic sinusoid', 'pericyte', 'adventitial cell', 'basal cell', 'neutrophil'
    .diseases = 'normal'
    .phenotypes = 'male', 'female'
    .experimental_factors = 'Smart-seq2', '10x 3' v3'
    .developmental_stages = '59-year-old human stage', '69-year-old human stage'
    .ethnicities = 'European'
    .ulabels = 'TSP14', 'TSP7', 'cell', 'tissue'
  Features
    'donor_id' = 'TSP14', 'TSP7'
    'suspension_type' = 'cell'
    'tissue_type' = 'tissue'
  Feature sets
    'obs' = 'assay', 'cell_type', 'development_stage', 'disease', 'donor_id', 'self_reported_ethnicity', 'sex', 'tissue', 'organism', 'tissue_type', 'suspension_type'
    'var' = 'RNF145', 'RNU4ATAC2P', 'LINC01847', 'ADRA1B', 'GAPDHP40', 'TTC1', 'FAM200C', 'None', 'LINC02159', 'GABRA1', 'LINC01202', 'CCNG1', 'HMMR', 'RN7SKP60', 'RPL21P59'

The database is populated correspondingly.

ln.view()
Hide code cell output
****************
* module: core *
****************
Artifact
uid version description key suffix accessor size hash hash_type n_objects n_observations visibility key_is_virtual storage_id transform_id run_id created_by_id updated_at
id
1 7JWPBDDaO3v4bsarcj50 2023-07-25 Tabula Sapiens - Salivary_Gland cell-census/2023-07-25/h5ads/f01bdd17-4902-40f... .h5ad AnnData 575121838 4DpycNCrKc_Q_pEYmPXZRw-69 md5-n None 27199 1 False 2 1 1 1 2024-05-25 15:25:34.578549+00:00
Feature
uid name dtype unit description synonyms run_id created_by_id updated_at
id
11 9s20Imh64bYU suspension_type cat[ULabel] None MUST be 'cell', 'nucleus', or 'na' None 1 1 2024-05-25 15:25:34.930441+00:00
10 69RQ1pVPEo3o tissue_type cat[ULabel] None MUST be 'tissue', 'organoid', or 'cell culture' None 1 1 2024-05-25 15:25:34.930317+00:00
9 wCbvLeKe2rv8 organism cat[bionty.Organism] None None None 1 1 2024-05-25 15:25:34.930193+00:00
8 MB1hKtGaQiqV tissue cat[bionty.Tissue] None None None 1 1 2024-05-25 15:25:34.930069+00:00
7 Xcti9JkM9uvG sex cat[bionty.Phenotype] None None None 1 1 2024-05-25 15:25:34.929944+00:00
6 DbVAV4BaQXAh self_reported_ethnicity cat[bionty.Ethnicity] None None None 1 1 2024-05-25 15:25:34.929821+00:00
5 xbMGSIMHV67J donor_id cat[ULabel] None None None 1 1 2024-05-25 15:25:34.929697+00:00
FeatureSet
uid name n dtype registry hash run_id created_by_id
id
1 VDzy5A1JdsCE3hxsU2TT None 11 None Feature FqF5x0w1rGBjZBGzh-Qr 1 1
2 G0zGHAaW2podIpvscSAb None 58604 number bionty.Gene vGM2j-OZYfQ4-9KsK75K 1 1
Run
uid started_at finished_at is_consecutive reference reference_type transform_id report_id environment_id created_by_id
id
1 UgJjwUTClVtI1VvlqyMt 2024-05-25 15:25:32.030280+00:00 None True None None 1 None None 1
Storage
uid root description type region instance_uid run_id created_by_id updated_at
id
2 oIYGbD74 s3://cellxgene-data-public None s3 us-west-2 None 1.0 1 2024-05-25 15:25:34.576239+00:00
1 p1aZOA3Bjv0c /home/runner/work/lamindb/lamindb/docs/test-tr... None local None 1FHu5eE0uxm4 NaN 1 2024-05-25 15:25:29.852616+00:00
Transform
uid version name key description type reference reference_type latest_report_id source_code_id created_by_id updated_at
id
1 ITeOtm7bhtdq5zKv 1 Transfer data transfer None notebook None None None None 1 2024-05-25 15:25:32.023966+00:00
ULabel
uid name description reference reference_type run_id created_by_id updated_at
id
4 orwn0HKG tissue None None None 1 1 2024-05-25 15:27:17.431915+00:00
3 WzKPMCXX cell None None None 1 1 2024-05-25 15:27:17.265179+00:00
2 1s9HrDbk TSP7 None None None 1 1 2024-05-25 15:27:17.098100+00:00
1 HVoqm0bp TSP14 None None None 1 1 2024-05-25 15:27:16.931077+00:00
User
uid handle name updated_at
id
1 00000000 anonymous None 2024-05-25 15:25:29.847453+00:00
******************
* module: bionty *
******************
CellType
uid name ontology_id abbr synonyms description public_source_id run_id created_by_id updated_at
id
77 1uUQwGAL endo-epithelial cell CL:0002076 None None An Epithelial Cell Derived From Endoderm. 29 1 1 2024-05-25 15:27:03.729755+00:00
76 6o2T6eYX epithelial cell of alimentary canal CL:0002251 None None An Epithelial Cell Of The Musculomembranous Di... 29 1 1 2024-05-25 15:27:03.262108+00:00
75 4cUGHo42 exocrine cell CL:0000152 None None A Cell Of An Exocrine Gland; I.E. A Gland That... 29 1 1 2024-05-25 15:27:03.261958+00:00
74 4eEkKmdU secretory cell CL:0000151 None None A Cell That Specializes In Controlled Release ... 29 1 1 2024-05-25 15:27:02.652125+00:00
73 7glBbUDw protein secreting cell CL:0000154 None None None 29 1 1 2024-05-25 15:27:02.076034+00:00
72 41jKWil4 glandular epithelial cell CL:0000150 None None A Specialized Epithelial Cell That Is Capable ... 29 1 1 2024-05-25 15:27:02.075850+00:00
71 dcsPbfKc salivary gland glandular cell CL:1001596 None None Glandular Cell Of Salivary Gland. Example: Ser... 29 1 1 2024-05-25 15:27:01.623636+00:00
DevelopmentalStage
uid name ontology_id abbr synonyms description public_source_id run_id created_by_id updated_at
id
3 7k1A0zkT human life cycle stage HsapDv:0000000 None developmental stage A Spatiotemporal Region Encompassing Some Part... 70 1 1 2024-05-25 15:27:15.109295+00:00
2 5q64GXx1 69-year-old human stage HsapDv:0000163 None None Adult Stage That Refers To An Adult Who Is Ove... 70 1 1 2024-05-25 15:27:14.519393+00:00
1 2zxrb1Pu 59-year-old human stage HsapDv:0000153 None None Middle Aged Stage That Refers To An Adult Who ... 70 1 1 2024-05-25 15:27:14.519239+00:00
Disease
uid name ontology_id abbr synonyms description public_source_id run_id created_by_id updated_at
id
1 6IZiH6X8 normal PATO:0000461 None None A Quality Inhering In A Bearer By Virtue Of Th... 63 1 1 2024-05-25 15:27:07.850027+00:00
Ethnicity
uid name ontology_id abbr synonyms description public_source_id run_id created_by_id updated_at
id
2 6Tnv21Sw ancestry category HANCESTRO:0004 None ancestral group Population Category Defined Using Ancestry Inf... 72 1 1 2024-05-25 15:27:16.492560+00:00
1 15mGvAtW European HANCESTRO:0005 None Caucasian|white Includes Individuals Who Either Self-Report Or... 72 1 1 2024-05-25 15:27:15.926255+00:00
ExperimentalFactor
uid name ontology_id abbr synonyms description molecule instrument measurement public_source_id run_id created_by_id updated_at
id
9 6JvKxplX 10x technology EFO:0008995 None 10x single cell library construction|10x Genomics 10X Is A "Synthetic Long-Read" Technology And ... None None None 51 1 1 2024-05-25 15:27:13.579064+00:00
8 1SAR2Y26 10x transcription profiling EFO:0030080 None None The 10X-Based Single-Cell Technology That Sequ... None None None 51 1 1 2024-05-25 15:27:13.083299+00:00
7 2kXleO7C 10x 3' transcription profiling EFO:0030003 None None 10X 3' Transcription Profiling Is The 10X-Base... None None None 51 1 1 2024-05-25 15:27:12.455710+00:00
6 1wLRxESw assay by molecule EFO:0002772 None None None None None None 51 1 1 2024-05-25 15:27:11.951426+00:00
5 5q5okCXl single cell library construction EFO:0010183 None single cell library preparation|single-cell li... A Library Preparation Process That Starts From... None None None 51 1 1 2024-05-25 15:27:11.327593+00:00
4 789nVHwo RNA assay EFO:0001457 None None An Assay With Input Rna RNA assay None None 51 1 1 2024-05-25 15:27:10.689961+00:00
3 1x8VaVN0 Smart-like EFO:0010184 None None Library Construction Method Using The Smart (S... None None None 51 1 1 2024-05-25 15:27:10.689778+00:00
Gene
uid symbol stable_id ensembl_gene_id ncbi_gene_ids biotype description synonyms organism_id public_source_id run_id created_by_id updated_at
id
58604 70hEAyHdv2mf None None ENSG00000272085 lncRNA novel transcript 1 13 1 1 2024-05-25 15:26:26.599285+00:00
58603 7fhcZHrnbaY1 None None ENSG00000253424 lncRNA novel transcript 1 13 1 1 2024-05-25 15:26:26.599118+00:00
58602 2QvvJOfXjXPd None None ENSG00000253449 lncRNA novel transcript 1 13 1 1 2024-05-25 15:26:26.598952+00:00
58601 6e9f3nDhfRWs None None ENSG00000253673 lncRNA novel transcript 1 13 1 1 2024-05-25 15:26:26.598783+00:00
58600 4ylvxaM60dXB None None ENSG00000253792 lncRNA novel transcript 1 13 1 1 2024-05-25 15:26:26.598616+00:00
58599 1gpu6hQGoHVj None None ENSG00000213414 processed_pseudogene cell division cycle 37 homolog (S. cerevisiae)... 1 13 1 1 2024-05-25 15:26:26.598449+00:00
58598 Z3GXFaLKPXSm RNU2-48P None ENSG00000222626 snRNA RNA, U2 small nuclear 48, pseudogene [Source:H... 1 13 1 1 2024-05-25 15:26:26.598281+00:00
Organism
uid name ontology_id scientific_name public_source_id run_id created_by_id updated_at
id
1 1dpCL6Td human NCBITaxon:9606 Homo sapiens 10 1 1 2024-05-25 15:25:40.616895+00:00
Phenotype
uid name ontology_id abbr synonyms description public_source_id run_id created_by_id updated_at
id
2 3OV8JZsS female PATO:0000383 None None A Biological Sex Quality Inhering In An Indivi... 63 1 1 2024-05-25 15:27:09.292311+00:00
1 kniZxLLf male PATO:0000384 None None A Biological Sex Quality Inhering In An Indivi... 63 1 1 2024-05-25 15:27:09.206964+00:00
PublicSource
uid entity organism currently_used source source_name version url md5 source_website run_id created_by_id updated_at
id
73 5JnV BioSample all True ncbi NCBI BioSample attributes 2023-09 s3://bionty-assets/df_all__ncbi__2023-09__BioS... 918db9bd1734b97c596c67d9654a4126 https://www.ncbi.nlm.nih.gov/biosample/docs/at... None 1 2024-05-25 15:25:29.959100+00:00
72 3Tlc Ethnicity human True hancestro Human Ancestry Ontology 3.0 https://github.com/EBISPOT/hancestro/raw/3.0/h... 76dd9efda9c2abd4bc32fc57c0b755dd https://github.com/EBISPOT/hancestro None 1 2024-05-25 15:25:29.958935+00:00
71 16tR DevelopmentalStage mouse True mmusdv Mouse Developmental Stages 2020-03-10 http://aber-owl.net/media/ontologies/MMUSDV/9/... 5bef72395d853c7f65450e6c2a1fc653 https://github.com/obophenotype/developmental-... None 1 2024-05-25 15:25:29.955641+00:00
70 7CRn DevelopmentalStage human True hsapdv Human Developmental Stages 2020-03-10 http://aber-owl.net/media/ontologies/HSAPDV/11... 52181d59df84578ed69214a5cb614036 https://github.com/obophenotype/developmental-... None 1 2024-05-25 15:25:29.955491+00:00
69 3TI0 Drug all False dron Drug Ontology 2023-03-10 https://data.bioontology.org/ontologies/DRON/s... 75e86011158fae76bb46d96662a33ba3 https://bioportal.bioontology.org/ontologies/DRON None 1 2024-05-25 15:25:29.955342+00:00
68 5alK Drug all True dron Drug Ontology 2024-03-02 https://data.bioontology.org/ontologies/DRON/s... 84138459de4f65034e979f4e46783747 https://bioportal.bioontology.org/ontologies/DRON None 1 2024-05-25 15:25:29.955193+00:00
67 3rm9 BFXPipeline all True lamin Bioinformatics Pipeline 1.0.0 s3://bionty-assets/bfxpipelines.json a7eff57a256994692fba46e0199ffc94 https://lamin.ai None 1 2024-05-25 15:25:29.955044+00:00
Tissue
uid name ontology_id abbr synonyms description public_source_id run_id created_by_id updated_at
id
17 7SvOepT5 buccal salivary gland UBERON:0012102 None None A Salivary Gland That Is Part Of A Cheek. 34 1 1 2024-05-25 15:26:42.450694+00:00
16 6TiLnvMv serous gland UBERON:0000409 None None A Gland In Which The Principal Secretory Cells... 34 1 1 2024-05-25 15:26:42.450544+00:00
15 2rlm5wUr anatomical entity UBERON:0001062 None None Biological Entity That Is Either An Individual... 34 1 1 2024-05-25 15:26:41.877519+00:00
14 N039zety material anatomical entity UBERON:0000465 None None Anatomical Entity That Has Mass. 34 1 1 2024-05-25 15:26:41.279855+00:00
13 5SGM2iq3 anatomical structure UBERON:0000061 None biological structure|connected biological stru... Material Anatomical Entity That Is A Single Co... 34 1 1 2024-05-25 15:26:40.787287+00:00
12 4o2HviGe multicellular anatomical structure UBERON:0010000 None multicellular structure An Anatomical Structure That Has More Than One... 34 1 1 2024-05-25 15:26:40.165307+00:00
11 7HJIkVT2 organ UBERON:0000062 None None Anatomical Structure That Performs A Specific ... 34 1 1 2024-05-25 15:26:39.665112+00:00
Hide code cell content
# clean up test instance
!lamin delete --force test-transfer
!rm -r test-transfer
❗ calling anonymously, will miss private instances
πŸ’‘ deleting instance anonymous/test-transfer
rm: cannot remove 'test-transfer': No such file or directory