Utilities¶
The command line utilities making up the SlyPI package are listed here. The available
command line options for any utility can be seen using the --help flag.
These utilities can also be called from within python, where the arguments are passed in a list.
Slycat Interaction¶
These utilities interact directly with Slycat, e.g. querying markings, adding/removing users, etc.
List markings¶
python -m slypi.util.list_markings --help
usage: list_markings.py [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY] [--https-proxy HTTPS_PROXY]
[--verify VERIFY] [--no-verify]
[--file-slice-size FILE_SLICE_SIZE] [--user USER]
[--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
Display available marking types supported by the Slycat server.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
List Projects¶
python -m slypi.util.list_projects --help
usage: list_projects.py [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY] [--https-proxy HTTPS_PROXY]
[--verify VERIFY] [--no-verify]
[--file-slice-size FILE_SLICE_SIZE] [--user USER]
[--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
List projects accessible for a given user.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
Get Project¶
python -m slypi.util.get_project --help
usage: Get details of project from Slycat server. [-h] --host HOST
[--port PORT]
[--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY]
[--verify VERIFY]
[--no-verify]
[--file-slice-size FILE_SLICE_SIZE]
[--user USER]
[--password PASSWORD]
[--kerberos]
[--log-level {debug,info,warning,error,critical}]
(--project-name PROJECT_NAME | --pid PID)
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--project-name PROJECT_NAME
Project name to retrieve.
--pid PID Project ID to retrieve.
Get Model¶
python -m slypi.util.get_model --help
usage: Get model details from Slycat server. [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY]
[--verify VERIFY] [--no-verify]
[--file-slice-size FILE_SLICE_SIZE]
[--user USER]
[--password PASSWORD]
[--kerberos]
[--log-level {debug,info,warning,error,critical}]
mid
positional arguments:
mid The ID of the model to retrieve
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
List Project Members¶
python -m slypi.util.list_project_members --help
usage: list_project_members.py [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY] [--verify VERIFY]
[--no-verify]
[--file-slice-size FILE_SLICE_SIZE]
[--user USER] [--password PASSWORD]
[--kerberos]
[--log-level {debug,info,warning,error,critical}]
(--project-name PROJECT_NAME | --pid PID)
List users for a given project.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--project-name PROJECT_NAME
Project name to retrieve.
--pid PID Project ID to retrieve.
Add Project Members¶
python -m slypi.util.add_project_members --help
usage: add_project_members.py [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY] [--verify VERIFY]
[--no-verify]
[--file-slice-size FILE_SLICE_SIZE]
[--user USER] [--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
(--project-name PROJECT_NAME [PROJECT_NAME ...] | --pid PID [PID ...] | --all-projects)
--project-users PROJECT_USERS
[PROJECT_USERS ...]
[--user-type {administrator,writer,reader}]
Add users to a given project. If a user is already on the project, that user's
role will be modified (e.g. reader to writer).
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--project-name PROJECT_NAME [PROJECT_NAME ...]
Project name to retrieve.
--pid PID [PID ...] Project ID to retrieve.
--all-projects Apply to all projects where you are an administrator.
--project-users PROJECT_USERS [PROJECT_USERS ...]
User names to add to project.
--user-type {administrator,writer,reader}
User type to add to project. Default is reader.
Remove Project Members¶
python -m slypi.util.rm_project_members --help
usage: rm_project_members.py [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY] [--verify VERIFY]
[--no-verify] [--file-slice-size FILE_SLICE_SIZE]
[--user USER] [--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
(--project-name PROJECT_NAME [PROJECT_NAME ...] | --pid PID [PID ...] | --all-projects)
--project-users PROJECT_USERS [PROJECT_USERS ...]
Remove users from a given project.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--project-name PROJECT_NAME [PROJECT_NAME ...]
Project name to retrieve.
--pid PID [PID ...] Project ID to retrieve.
--all-projects Apply to all projects where you are an administrator.
--project-users PROJECT_USERS [PROJECT_USERS ...]
User names to add to project.
Uploading Slycat Models¶
These utilities allow users to upload Slycat models.
Parameter Space¶
python -m slypi.ps.upload_csv --help
usage: upload_csv.py [-h] --host HOST [--port PORT] [--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY] [--verify VERIFY]
[--no-verify] [--file-slice-size FILE_SLICE_SIZE]
[--user USER] [--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
[--input-columns [INPUT_COLUMNS ...]]
[--output-columns [OUTPUT_COLUMNS ...]]
[--categorical-columns [CATEGORICAL_COLUMNS ...]]
[--xy-pair XY_PAIR [XY_PAIR ...]]
[--media-columns [MEDIA_COLUMNS ...]]
[--media-hostname MEDIA_HOSTNAME] [--strip STRIP]
[--marking MARKING]
[--model-description MODEL_DESCRIPTION]
[--model-name MODEL_NAME]
[--project-description PROJECT_DESCRIPTION]
(--project-name PROJECT_NAME | --project-id PROJECT_ID)
[--debug]
file
Create Slycat Parameter Space model from .csv table.
positional arguments:
file Input CSV file.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--input-columns [INPUT_COLUMNS ...]
Input column names, can't be media columns.
--output-columns [OUTPUT_COLUMNS ...]
Output column names, can't be media columns.
--categorical-columns [CATEGORICAL_COLUMNS ...]
Categorical column names, can't be media coluns.
--xy-pair XY_PAIR [XY_PAIR ...]
Column pairs to be tagged using [XYpair] notation for
Slycat. Note: pairs should be of the form "<name>
X,<name> Y".
--media-columns [MEDIA_COLUMNS ...]
Columns that contain media URLs.
--media-hostname MEDIA_HOSTNAME
Optionally override the hostname for media columns.
--strip STRIP Optionally strip N prefix directories from the stored
URLs.
--marking MARKING Marking type. Default: mna
--model-description MODEL_DESCRIPTION
New model description. Default:
--model-name MODEL_NAME
New model name. Default: SlyPI PS Model
--project-description PROJECT_DESCRIPTION
New project description. Default:
--project-name PROJECT_NAME
Project name (will create if doesn't exist).
--project-id PROJECT_ID
Project ID.
--debug
CCA¶
python -m slypi.cca.upload_csv --help
usage: upload_csv.py [-h] --host HOST [--port PORT] [--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY] [--verify VERIFY]
[--no-verify] [--file-slice-size FILE_SLICE_SIZE]
[--user USER] [--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
[--project-description PROJECT_DESCRIPTION]
--project-name PROJECT_NAME [--model-name MODEL_NAME]
[--model-description MODEL_DESCRIPTION]
[--input-columns INPUT_COLUMNS [INPUT_COLUMNS ...]]
[--output-columns OUTPUT_COLUMNS [OUTPUT_COLUMNS ...]]
[--marking MARKING] [--scale-inputs] [--no-join]
file
Create CCA model from .csv file.
positional arguments:
file Input CSV file. Use - for stdin. Default: -
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--project-description PROJECT_DESCRIPTION
New project description. Default:
--project-name PROJECT_NAME
New project name.
--model-name MODEL_NAME
New model name. Default: SlyPI CCA Model
--model-description MODEL_DESCRIPTION
New model description. Default:
--input-columns INPUT_COLUMNS [INPUT_COLUMNS ...]
Input column(s).
--output-columns OUTPUT_COLUMNS [OUTPUT_COLUMNS ...]
Output column(s).
--marking MARKING Marking type. Default: mna
--scale-inputs Enable input scaling.
--no-join Don't wait for the model to finish.
Dial-A-Cluster¶
python -m slypi.dac.upload_gen --help
usage: upload_gen.py [-h] --host HOST [--port PORT] [--http-proxy HTTP_PROXY]
[--https-proxy HTTPS_PROXY] [--verify VERIFY]
[--no-verify] [--file-slice-size FILE_SLICE_SIZE]
[--user USER] [--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
[--marking MARKING]
[--model-description MODEL_DESCRIPTION]
[--model-name MODEL_NAME]
[--project-description PROJECT_DESCRIPTION]
--project-name PROJECT_NAME
dac_gen_zip
Create Slycat Dial-A-Cluster model from dac generic .zip file.
positional arguments:
dac_gen_zip DAC generic .zip file.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--marking MARKING Marking type. Default: mna
--model-description MODEL_DESCRIPTION
New model description. Default:
--model-name MODEL_NAME
New model name. Default: SlyPI DAC Model
--project-description PROJECT_DESCRIPTION
New project description. Default:
--project-name PROJECT_NAME
New project name.
VideoSwarm¶
python -m slypi.vs.upload_local --help
usage: upload_local.py [-h] --host HOST [--port PORT]
[--http-proxy HTTP_PROXY] [--https-proxy HTTPS_PROXY]
[--verify VERIFY] [--no-verify]
[--file-slice-size FILE_SLICE_SIZE] [--user USER]
[--password PASSWORD] [--kerberos]
[--log-level {debug,info,warning,error,critical}]
--csv-file CSV_FILE --xcoords-file XCOORDS_FILE
--ycoords-file YCOORDS_FILE --traj-file TRAJ_FILE
--video-column VIDEO_COLUMN
[--video-hostname VIDEO_HOSTNAME] [--strip STRIP]
[--marking MARKING]
[--model-description MODEL_DESCRIPTION]
[--model-name MODEL_NAME]
[--project-description PROJECT_DESCRIPTION]
--project-name PROJECT_NAME
Create Slycat VideoSwarm model from local format (.csv, .xcoords, .ycoords,
.trajectories) files.
options:
-h, --help show this help message and exit
--host HOST Root URL of the Slycat server.
--port PORT Port of the Slycat server.
--http-proxy HTTP_PROXY
HTTP proxy URL. Default: ''.
--https-proxy HTTPS_PROXY
HTTPS proxy URL. Default: ''.
--verify VERIFY Specify a certificate to use for HTTPS host
certificate verification.
--no-verify Disable HTTPS host certificate verification.
--file-slice-size FILE_SLICE_SIZE
Maximum number of bytes to upload before slicing files
(i.e. before uploading in chunks). Default: 10000000.
--user USER Slycat username. Default: 'docs'
--password PASSWORD User password.
--kerberos Use Kerberos authentication. Default: False.
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'.
--video-hostname VIDEO_HOSTNAME
Optionally override the hostname for the video column.
--strip STRIP Optionally strip N prefix directories from the video
URLs.
--marking MARKING Marking type. Default: mna
--model-description MODEL_DESCRIPTION
New model description. Default:
--model-name MODEL_NAME
New model name. Default: SlyPI VS Model
--project-description PROJECT_DESCRIPTION
New project description. Default:
--project-name PROJECT_NAME
New project name.
required arguments:
--csv-file CSV_FILE Input .csv file.
--xcoords-file XCOORDS_FILE
Input .xcoords file.
--ycoords-file YCOORDS_FILE
Input .ycoords file.
--traj-file TRAJ_FILE
Input .trajectories file.
--video-column VIDEO_COLUMN
Header name of video column.
Ensemble Manipulation¶
These utilities are used to read and manipulate the files making up ensemble data, typically from numerical simulations. They will accept various file formats and output various Slycat file formats. They can be extended to work with new file formats using plugins.
table.py¶
python -m slypi.ensemble.table --help
usage: table.py [-h] [--log-level {debug,info,warning,error,critical}]
[--log-file LOG_FILE] [--plugin PLUGIN]
[--create | --join JOIN [JOIN ...] | --concat CONCAT
[CONCAT ...] | --expand EXPAND | --convert-uris CONVERT_URIS |
--convert CONVERT] [--ensemble ENSEMBLE]
[--input-files INPUT_FILES] [--input-header INPUT_HEADER]
[--input-format INPUT_FORMAT] [--ignore-index] [--no-index]
[--uri-cols URI_COLS [URI_COLS ...]] [--uri-root URI_ROOT]
[--add-origin-col ADD_ORIGIN_COL]
[--origin-col-names ORIGIN_COL_NAMES [ORIGIN_COL_NAMES ...]]
[--expand-header EXPAND_HEADER] [--output-dir OUTPUT_DIR]
[--output-file OUTPUT_FILE] [--output-no-index]
[--output-index-header OUTPUT_INDEX_HEADER]
[--output-headers [OUTPUT_HEADERS ...]]
[--exclude-output-headers [EXCLUDE_OUTPUT_HEADERS ...]]
[--over-write]
Creates/manipulates files from ensemble data set.
options:
-h, --help show this help message and exit
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'
--log-file LOG_FILE Log to file. Notes: (1) If this file already exists it
will be overwritten, (2) Log file includes time stamp
and is set to debug level
--plugin PLUGIN Plugin Python file name to import (defaults to 'ps'),
can be either a plugin from slypi/ensemble/plugins (no
extension) or a python file (.py extension). Use
"python -m slypi.ensemble.plugins.plugin --help" to
see any command line options for the plugin.
--create Create ensemble file from simulation input decks.
--join JOIN [JOIN ...]
List of slypi.ensemble files to join horizontally
(first column is assumed to be index).
--concat CONCAT [CONCAT ...]
List of slypi.ensemble files to join vertically (all
column headers must be identical).
--expand EXPAND Expand links in input file to include data in table.
Uses plugin to expand links.
--convert-uris CONVERT_URIS
Converts columns from the given table to URIs, columns
are specified using --uri-cols, and conversion is
specified using --uri-root.
--convert CONVERT Convert a slypi.ensemble table to a final Slycat table
(remove source index).
--ensemble ENSEMBLE Directory or directories to include in ensemble,
specified using the Python like %d[::] notation
described above.
--input-files INPUT_FILES
Files per ensemble directory to use as input for
metadata, specified using %d[::] notation. Note that
these files will be pre-fixed by the ensemble
directories.
--input-header INPUT_HEADER
Name to assign input file header, e.g. "Input Deck"
--input-format INPUT_FORMAT
Format for input files. Optional, inferred from file
extension if not provided.
--ignore-index Ignore index column when joining tables.
--no-index No index present in input file, join rows without
error checking (note that no output index is written,
over-riding --no-output-index).
--uri-cols URI_COLS [URI_COLS ...]
Columns with file points to convert to URIs.
--uri-root URI_ROOT Root name of URI used to transform file pointers in
output file when joining files. Note that this will
only work if the file pointers have a common root. All
paths in this case will be converted to unix paths.
--add-origin-col ADD_ORIGIN_COL
Add a column containing the data origin. This flag
gives the new column name.
--origin-col-names ORIGIN_COL_NAMES [ORIGIN_COL_NAMES ...]
Names to use for origin column, one per file to
concatenate (defaults to file names).
--expand-header EXPAND_HEADER
Table column to expand (either name or index).
--output-dir OUTPUT_DIR
Output directory for any files produced.
--output-file OUTPUT_FILE
File name of output file.
--output-no-index Do not output the index column.
--output-index-header OUTPUT_INDEX_HEADER
Index header name for output file (default is None).
--output-headers [OUTPUT_HEADERS ...]
Output only the given headers to the output file
(defaults to all headers).
--exclude-output-headers [EXCLUDE_OUTPUT_HEADERS ...]
Exclude the given headers from the output file.
--over-write Over-write output file if already present.
The table utilities can also be called directly using a python API. Each of the major functions is split out as described below.
- slypi.ensemble.table.join_csv(join_tables, ensemble_spec=None, input_files=None, input_header=None, ignore_index=False, no_index=False, output_dir=None, output_file=None, output_no_index=None, output_index_header=None, output_headers=None, exclude_output_headers=None, log=None)¶
Joins files containing tables, either slypi intermediate format (with an index column), or standard CSV tables. The arguments are all optional, supporting different ways to combine tables. Further descriptions of the arguments and their interactions can be found using table.py –help (under the –join option).
- Parameters:
join_tables (list) – file names of tables to join
ensemble_spec (string) – directory or directories to include in ensemble.
input_files (string) – files per ensemble directory to use as input.
input_header (string) – name to assign input file header in table
ignore_index (bool) – ignore index (first) column when joining tables
no_index (bool) – join tables without using index (first) column
output_dir (string) – directory to output combined table
output_file (string) – output file name (extension agnostic)
output_no_index (bool) – do not output index column (if not using ignore_index)
output_index_header (string) – index header name
output_headers (list) – list of strings of headers to output
exclude_output_headers (list) – list of strings of headers to exclude
log (object) – logger function, if not supplied will output to screen
- Returns:
Writes a combined table to the requested file.
Note: Does not check for over-writes or existence of directories.
- Example:
from slypi.ensemble.table import join_csv join_csv (join_tables=[os.path.join(test_data_dir, 'metadata.csv'), os.path.join(convert_dir, 'end-state.csv'), os.path.join(convert_dir, 'movies.csv')], output_headers=['mobility_coefficients-1', 'mobility_coefficients-2', 'composition_distribution-1', 'End State', 'Movie'], output_dir=output_dir, output_file='ps-no-index-api.csv', no_index=True)
- slypi.ensemble.table.convert_uris(table_csv, uri_cols=None, uri_root=None, output_dir=None, output_file=None, output_headers=None, exclude_output_headers=None, log=None)¶
Converts file pointers in a table to URIs given a URI root name.
- Parameters:
table_csv (string) – file name of table to use
uri_cols (list) – list of header names to convert in table
uri_root (string) – uri root path for conversion
output_dir (string) – directory to output combined table
output_file (string) – output file name (extension agnostic)
output_headers (list) – list of strings of headers to output
exclude_output_headers (list) – list of strings of headers to exclude
log (object) – logger function, if none supplied will output to screen
- Returns:
Writes a new table with the URI conversions.
Note: Does not check for over-writes or existence of directories.
- Example:
from slypi.ensemble.table import convert_uris convert_uris(os.path.join(output_dir, 'ps-no-index.sly'), uri_cols=['End State', 'Movie'], uri_root=uri_root_out, output_dir=output_dir, output_file='ps-uri-convert-api.csv')
convert.py¶
python -m slypi.ensemble.convert --help
usage: convert.py [-h] [--log-level {debug,info,warning,error,critical}]
[--log-file LOG_FILE] [--plugin PLUGIN]
[--ensemble ENSEMBLE] [--input-files INPUT_FILES]
[--csv-file CSV_FILE] [--csv-col CSV_COL]
[--input-format INPUT_FORMAT] [--output-dir OUTPUT_DIR]
[--output-format OUTPUT_FORMAT] [--over-write]
[--csv-out CSV_OUT] [--csv-header CSV_HEADER] [--parallel]
Converts ensemble file formats. Uses Python-like %d[::] notation, where %d[::]
specifies a range of numbers in a file name. For example
"time_step_%d[0:10:2].vtp" would specify every other file from
"time_step_0.vtp" to "time_step_9.vtp".
options:
-h, --help show this help message and exit
--log-level {debug,info,warning,error,critical}
Log level. Default: 'info'
--log-file LOG_FILE Log to file. Notes: (1) If this file already exists it
will be overwritten, (2) Log file includes time stamp
and is set to debug level
--plugin PLUGIN Plugin Python file name to import (defaults to 'ps'),
can be either a plugin from slypi/ensemble/plugins (no
extension) or a python file (.py extension). Use
"python -m slypi.ensemble.plugins.plugin --help" to
see any command line options for the plugin.
--ensemble ENSEMBLE Directory or directories to include in ensemble,
specified using the Python like %d[::] notation
described above.
--input-files INPUT_FILES
Files per ensemble directory to use as input for
conversion, specified using %d[::] notation. Note that
these files will be pre-fixed by the ensemble
directories.
--csv-file CSV_FILE CSV file which specifies ensemble directories and
input files (alternate to using --ensemble and
--input-files).
--csv-col CSV_COL Column in CSV file where input files are specified,
can be either a string or an integer (1-based).
--input-format INPUT_FORMAT
Format for input files (optional, inferred from file
extension if not provided).
--output-dir OUTPUT_DIR
Directory to place output. All files will be stored
using directories that mirror those specified by
--ensemble.
--output-format OUTPUT_FORMAT
File output format information, e.g. file extension.
--over-write Over-write output directory if already present.
--csv-out CSV_OUT File name of output .csv file with file links for
converted files (optional). Will be written to output
directory.
--csv-header CSV_HEADER
Name of output files header, needed only if writing
out a .csv file.
--parallel Use ipyparallel (must be available and running).
