Data import functionality

[yassene]

Hello everyone,

I was just wondering about the data import functionality. I thought we can start to collect ideas about how to do that for each modules. In my opinion this will be important for usability. Ideas?

[ergonyc]

Thanks for kicking this off @yassene ! I think that we need to decide on a couple of "formats" to support, and develop a simple UI to automate the "curation". There are a couple of helper functions (ie. omicser::setupe_database found in pre_process_helpers.R ) and examples in the curation scripts of what needs to be done. Scanpy, Seurat, and tabular data examples are what we have worked through so far.

THe next step is to refine the schema of what we will require in a dataset to make it usefully browsed.

Some of the functionality in the "ingest" tab is a rough attempt in this direction, but I think a clean modal, or hidden tab to do this could be great.

[ddifraia]

A direct idea that comes into my mind, is to take in input fundamentally 3 different kind of files

• A matrix (n x m) where n = proteins, m= samples. (in this case we can also think to have attached some features metadata)
• A sample feature, again a matrix (m x z) where m = samples and z = samples metadata, (condition, replicates, etc etc...)
• Depending on the dataset (?) a differential expression table, with logFC values, pvalues/qvalues/ conditions,

If I think we can abstract from this more or less for all kinds of -omics data. The user would need them to give the data in this format, that I think is rather generic.

We could 'force' the user to select certain column that must be there.
For the mapping between different dataset, we could think to annotate the data internally, (adding mapping information, based on the type of entries that one user has in his matrix, es, UNIPROT, GENE SYMBOL, ENSEML.. etc etc..).

Or alternatively the user could specify a column in its features metadata that could be used for cross-mapping with different datasets.

Let me know what do you think

[ricoderks]

Maybe a small remark @ddifraia. I see that you describe the data matrix as proteins x samples. I'am use to observations x variables, this would mean samples x proteins. I think this is also the standard in (multivariate) statistics.

[ddifraia]

Yes, @ricoderks you are definitely right, I was just reasoning in terms of proteins and matrix that we have as an example (we are used in the lab to this kind of format). For the multivariate statistics, depends of what analysis you want to run. In the end depending on the different needs one would just need probably to transpose the matrix.

[ricoderks]

I've been working on a GUI for the data curation. It is not completely finished yet. For now it only contains a lipidomics part (in progress). @ergonyc is it possible to create a separate branch of the main-dev branch where I can put the work I have so far? I put a button next to the add database button and then it will show a modal.

I created a module for the general part, i.e. button and show modal. In here we can add several modules each specific for an omics field. For uploading the data set I used what @ddifraia mentioned above. Three files should be uploaded:

• data
• observations (samples)
• variables (lipids)

[ddifraia]

Super cool! :D

[ddifraia]

Probably here, we should consider the possibility to ingest a differential expression table pre-calculated by some other software (example DESeq2 for Transcriptomic, and our MS-Search software for Proteomics) as well. In a similar fashion the user could select in which column of the data frame the differential expression and values lives, to use the data for a volcano etc...etc...

[ricoderks]

This should be easily added to the curation part. @yassene and I where thinking if it is maybe not nicer to calculate the differential expression on the fly, so depending on which grouping information you choose. Only transcriptomics data will be a bit slow than. I planned to see if the package data.table can speed things up.

[ricoderks]

For lipidomics I pushed a first version (needs a little bit of cleanup) for doing curation from omicser. It is in the branch main-dev_curation.

Needed:

• A csv file with the data_matrix. 1st column contains sample id's (sample_id), the other column names are the names of lipids given by the lipidyzer.
• A csv file with the sample info. 1st column contains sample id's (sample_id) and it can contain additional columns with grouping information.
• The lipid information is extracted from a master table containing all lipids which can be measured with the lipidyzer and this master table also contains some linking information.

Todo:

• It now checks only if a database name exists instead of the folder. Need to change this to folder.
• In the feature details it shows NA if information is not available, need to make this a bit nicer.

Example files:
conc_data_matrix.csv
sample_info.csv