How to set up and use the validation in EDAMIS Project Author Release Status Reference January 2011 EDAMIS Eurostat, Statistical Information Technologies, Unit B5 1.0 Draft QUG-05-2011-02-07 [QUG] (How to set up and use the validation in EDAMIS) D.doc Page 1 TABLE OF CONTENTS 1 INTRODUCTION ......................................................................................................................................... 5 1.1 2 BACKGROUND INFORMATION .................................................................................................................... 5 GETTING STARTED .................................................................................................................................. 7 2.1 2.2 3 WHAT VALIDATIONS ARE .......................................................................................................................... 7 GENERAL PREREQUISITES .......................................................................................................................... 7 CREATING VALIDATIONS ...................................................................................................................... 8 3.1 ACCESSING THE VALIDATION MANAGEMENT FORM ................................................................................... 8 3.2 LINKING A DSD TO A DATASET ................................................................................................................. 9 3.3 CREATING A VALIDATION RULES SET....................................................................................................... 10 3.4 CREATING A VALIDATION RULE ............................................................................................................... 12 3.4.1 Defining the scope of a validation rule ......................................................................................... 14 3.4.2 Defining a validation rule ............................................................................................................. 15 3.4.2.1 Normal rules ........................................................................................................................ 16 3.4.2.2 Advanced rules .................................................................................................................... 17 4 TESTING A VALIDATION RULES SET ............................................................................................... 20 4.1 5 ANALYZING A VALIDATION REPORT ........................................................................................................ 21 PUBLISHING VALIDATIONS................................................................................................................. 22 5.1 6 VALIDATION MECHANISM........................................................................................................................ 22 MANAGING VALIDATIONS .................................................................................................................. 24 6.1 6.2 6.3 7 LOCKING A VALIDATION .......................................................................................................................... 24 UNLOCKING A VALIDATION ..................................................................................................................... 25 DELETING A VALIDATION ........................................................................................................................ 26 MANAGING VALIDATION RULES SETS ............................................................................................ 28 7.1 7.2 8 COPYING A VALIDATION RULES SET......................................................................................................... 28 DELETING A VALIDATION RULES SET ....................................................................................................... 28 MANAGING VALIDATION RULES ...................................................................................................... 30 8.1 8.2 9 EDITING A VALIDATION RULE .................................................................................................................. 30 REMOVING A VALIDATION RULE .............................................................................................................. 30 VALIDATING DATA FILES .................................................................................................................... 31 9.1 SENDING DATA FILES THROUGH EWP ..................................................................................................... 31 9.1.1 Sending data files containing critical errors ................................................................................. 32 9.1.2 Sending data files containing basic errors .................................................................................... 33 9.1.3 Sending correct data files .............................................................................................................. 34 9.1.4 Validation reports availability ...................................................................................................... 35 9.2 SENDING DATA FILES THROUGH EWA ..................................................................................................... 35 10 ANNEX .................................................................................................................................................. 37 11 GLOSSARY .......................................................................................................................................... 48 12 HISTORY OF CHANGES ................................................................................................................... 49 January 2011 Page 2 TABLE OF FIGURES Figure 1: Manage Validation menu entry ............................................................................................................... 8 Figure 2: Manage Validation form.......................................................................................................................... 8 Figure 3: Manage Validation form with no existing validation .............................................................................. 9 Figure 4: Error message when willing to associate a DSD ..................................................................................... 9 Figure 5: Manage Validation - Associate a DSD form ........................................................................................... 9 Figure 6: Warning message when associating a DSD ........................................................................................... 10 Figure 7: Error message when associating a DSD ................................................................................................ 10 Figure 8: Manage Validation form with the newly created validation .................................................................. 10 Figure 9: Empty Manage Validation – Data Structure Definition form ............................................................... 11 Figure 10: Manage Validation – Add a rule set to the DSD form ......................................................................... 12 Figure 11: Manage Validation – DSD form with one validation rules set ............................................................ 12 Figure 12: Validation rules editor ......................................................................................................................... 13 Figure 13: Excerpt of the DSD used throughout the manual ................................................................................ 14 Figure 14: Validation rules editor – Specifying a dimension by label .................................................................. 15 Figure 15: Validation rules editor – Specifying a dimension by code .................................................................. 15 Figure 16: Normal validation rule popup window ................................................................................................ 16 Figure 17: Advanced validation rule popup window ............................................................................................ 18 Figure 18: Referencing another observation value................................................................................................ 19 Figure 19: Manage Validation – Test rules form .................................................................................................. 20 Figure 20: Validation report after testing the validation process .......................................................................... 20 Figure 21: Manage Validation form with the newly published validation ............................................................ 22 Figure 22: Confirmation message when clicking on a validation entry ................................................................ 24 Figure 23: Confirmation message when clicking on a published validation entry ................................................ 24 Figure 24: Error message when locking a validation ............................................................................................ 25 Figure 25: Error message when locking an already locked validation .................................................................. 25 Figure 26: Validation listed as locked ................................................................................................................... 25 Figure 27: Error message when unlocking a validation ........................................................................................ 26 Figure 28: Error message when locking an already locked validation .................................................................. 26 Figure 29: Validation listed as not locked ............................................................................................................. 26 Figure 30: Confirmation message when deleting validations ............................................................................... 27 Figure 31: Error message when deleting validations ............................................................................................ 27 Figure 32: Confirmation message when deleting validations ............................................................................... 27 Figure 33: Manage Validation – Add a rule set to the DSD form ......................................................................... 28 Figure 34: Confirmation message when deleting validation rules sets ................................................................. 28 Figure 35: Confirmation message when deleting validation rules sets ................................................................. 29 Figure 36: Send Datafile menu entry .................................................................................................................... 31 Figure 37: Send Datafile form when selecting a data file to be validated ............................................................. 32 Figure 38: Uploading a data file containing a (critical) error (Advanced upload form) ....................................... 32 Figure 39: Uploading a data file containing a (critical) error (Simple upload form) ............................................ 32 Figure 40: Send Datafile form after data files upload ........................................................................................... 33 Figure 41: Validation report with a (critical) error................................................................................................ 33 Figure 42: Uploading a data file containing a warning (basic error) (Advanced upload form) ............................ 33 Figure 43: Uploading a data file containing a warning (basic error) (Simple upload form) ................................. 33 Figure 44: Validation report with a warning (basic error) .................................................................................... 34 Figure 45: Uploading a data file with no error (Advanced upload form) .............................................................. 34 Figure 46: Uploading a data file with no error (Simple upload form) .................................................................. 34 Figure 47: Validation report with no error ............................................................................................................ 35 Figure 48: EWA - Send Datafile menu entry ........................................................................................................ 35 Figure 49: EWA - Send Datafiles form when selecting a data file to be validated ............................................... 35 Figure 50: EWA - Send Datafiles form after data files upload ............................................................................. 36 January 2011 Page 3 TABLE OF TABLES Table 1: Manage Validation form – Selection area .............................................................................................. 37 Table 2: Manage Validation form – Validations description area ........................................................................ 38 Table 3: Manage Validation form – Action buttons ............................................................................................. 38 Table 4: Manage Validation - Associate a DSD form – Definition area ............................................................... 39 Table 5: Manage Validation - Associate a DSD form – Action buttons ............................................................... 39 Table 6: Manage Validation – DSD Form form – Summary area ........................................................................ 40 Table 7: Manage Validation – DSD Form form – Validation rules sets description area ..................................... 40 Table 8: Manage Validation – DSD Form form – Action buttons ........................................................................ 41 Table 9: Manage Validation - Add a rules set to the DSD form – Definition area ............................................... 42 Table 10: Manage Validation - Add a rules set to the DSD form – Action buttons .............................................. 42 Table 11: Validation rules editor – Selection area ................................................................................................ 42 Table 12: Validation rules editor – Validation rules description area .................................................................. 43 Table 13: Validation rules editor – Action buttons ............................................................................................... 44 Table 14: Validation rules editor – Normal Rule popup window ......................................................................... 45 Table 15: Validation rules editor – Advanced Rule popup window...................................................................... 46 Table 16: Manage Validation - Create a copy of the rules set form – Definition area ......................................... 46 Table 17: Manage Validation - Create a copy of the rules set form – Action buttons.......................................... 47 Table 18: Manage Validation - Test rules form – Definition area ........................................................................ 47 Table 19: Manage Validation - Test rules form – Action buttons ........................................................................ 47 January 2011 Page 4 1 Introduction This manual is intended for the domain managers (DMA) who are willing to use the EDAMIS Validation Engine (EVE) to validate the data files that are transferred to Eurostat. It contains everything a domain manager needs to know in order to establish validations. It illustrates how to create validations and their components, how to maintain them and how to publish them. The validation process is also described in this manual. This manual is being made available to Eurostat production units to assist the implementation phase of the validations. 1.1 Background information EDAMIS: the electronic Data file Administration and Management Information System is the integrated environment for data transmission and for the Single Entry Point for data arriving at Eurostat. It is visible through the following tools: EWA: The EDAMIS Web Application is installed in the National Statistical Institutes and used for sending statistical data files to Eurostat. Interactive, semi-automated and fullautomated transmissions are supported. EWP: The EDAMIS Web Portal is installed at the European Commission and used for managing the dataset inventory, managing the user rights related to the transmissions, sending statistical data files to Eurostat (as an alternative to EWA) and monitoring the traffic through its Management Information System. EWF: The EDAMIS Web Form is a component of the EDAMIS Web Portal that can be used for the transmission of low volumes of data. It offers all the basic functions of EWP and does not require any installation on the PC of the user. EDAMIS Web Forms are typically of interest for the collection of data in statistical domains where: • data volumes are low • currently collection of data takes place via spreadsheet based questionnaires • a certain level of validation of the data before transmission is preferred • software installation on client PCs is forbidden • a data sharing agreement with international organisations exist Local Coordinators: In each National Statistical Institute (NSI), a Local Coordinator (LC) is the main contact point to Eurostat for the effective implementation of electronic data transmission and a contact point for members of their own organisation. Transmission Coordinators: In each National Statistical Institute (NSI), a Transmission Coordinator (TCO) is responsible for the management of users' rights within EDAMIS. The TCO is in charge of managing and giving rights to users inside its organisation. In NSIs the TCO is usually the Local Coordinator, while in non NSIs, a TCO can be nominated. A TCO January 2011 Page 5 can be allowed to coordinate users in several organisations of the country, provided that an agreement exists between the organisations concerned. Data sender and EDAMIS: In each National Statistical Institutes, the installed EWA allows data provider to send data to Eurostat. If the data sender is located in another organisation, then the EWP will be used instead. EDAMIS Web Forms, when they exist, can be obtained via the EWP. In all cases, data sender should have a CIRCA user-id and should register to EDAMIS. Further details for EDAMIS and CIRCA registration are available at the following URL: http://circa.europa.eu/irc/dsis/EDAMIS/info/data/website/index.htm January 2011 Page 6 2 Getting started 2.1 What validations are The aim of the validation process is to verify the consistency of the data files sent via EDAMIS, the Eurostat Single Entry Point (SEP) for data. This encompasses both the structure of the files as well as the data themselves. The validation process can take place on all EDAMIS transmission channels: EDAMIS Web Portal (EWP) and EDAMIS Web Application (EWA). Validations are specific to a dataset and are managed by a Eurostat statistical Domain Manager (DMA). The validation engine is based on a world wide standard for exchanging statistical data and metadata: SDMX-ML. For validation of data in other formats, i.e. CSV, tools like the SDMX Converter can be used before uploading the data to EDAMIS. The validation of the data file structure is based on a SDMX Data Structure Definitions (DSD) file which is associated to the dataset for which the data file is sent. The validation of the data is based on the validation rules defined by the DMA for the aforementioned relationship. 2.2 General prerequisites To be able to send data files to Eurostat, a domain and a dataset have to be defined in EWP. The transmission mode of the dataset has to be set either to Send data or to All as the validation process will not be triggered for Web Form entries. Even if the validations can be managed while the dataset is not active, the dataset will have to be activated in order for the users to be able to send data files. To be able to manage validations, the user needs Domain Manager (DMA) or Domain CoResponsible (DCR) right in EDAMIS for the domain related to the dataset. The corresponding DSD must exist in the SDMX Registry (https://webgate.training.ec.europa.eu/sdmxregistry) for EWP (and later on the validation engine) to retrieve it. As the definition of the validations will be done through an applet, Java has to be installed on the user’s PC. January 2011 Page 7 3 Creating validations The reason why the validation process relies on the usage of a DSD is because the purpose of a DSD file is to describe how the related data have to be structured when they are transferred to Eurostat for a given dataset. As such, the very first step when creating a validation consists of linking a DSD to the dataset for which the data files have to be validated. Once the link is made, the structure of the transferred data files can already be validated against the DSD providing that the validation is published (see paragraph ‘Publishing validations’). Furthermore, if custom data validation is required, the user can define one or several sets of validation rules in a second step. 3.1 Accessing the validation management form Once connected to the EDAMIS Web Portal, the user has to select the Inventory >> Manage Validation menu entry (see Figure 1) to access the validation management form. Figure 1: Manage Validation menu entry In the appearing Manage Validation form (see Figure 2), the user has first to filter the theme, domain and dataset for which he wants to create the validation and then click on the [View] button (see Table 1). Figure 2: Manage Validation form Once refreshed (see Figure 3), the form lists the existing validations if any (see Table 2) and allows the DMA to perform several actions (see Table 3). January 2011 Page 8 Figure 3: Manage Validation form with no existing validation 3.2 Linking a DSD to a dataset To link a DSD to the selected dataset, the DMA has to click on the [Associate a dsd] button. If the filters have not been correctly set, the following error message will appear: Figure 4: Error message when willing to associate a DSD Otherwise, on the appearing form (see Figure 5), the DMA has to provide the ID, the version and the agency of the DSD he wants to associate to the dataset (see Table 4). A link allows him to connect to the SDMX Registry (using his CIRCA credentials) in order to find the appropriate values. Figure 5: Manage Validation - Associate a DSD form Once the fields have been provided, the DMA has to click on the [Associate] button to confirm his choice (see Table 5). If the DSD is not found in the SDMX Registry, the following warning message will appear: January 2011 Page 9 Figure 6: Warning message when associating a DSD As a given DSD can be linked only once to a given dataset, the following error message will appear if the DSD has been already associated to the current dataset: Figure 7: Error message when associating a DSD In both cases, the DMA has the possibility to modify the fields or to abort the association by clicking on the [Cancel] button. Otherwise, if the DSD can be successfully retrieved from the SDMX Registry, the validation is then created and listed as locked by the DMA in the Manage Validation form (see Figure 8). Other properties of the validation are also summarized (see Table 2). Figure 8: Manage Validation form with the newly created validation Now that the link is made, the structure of the transferred data files can already be validated against the DSD providing that the validation is published (see paragraph ‘Publishing validations’). 3.3 Creating a validation rules set To be able to create validation rules, the DMA has to create beforehand a validation rules set that will act as a container for the validation rules. January 2011 Page 10 As a given DSD can be linked only once to a given dataset, the DMA can keep track of different versions of the validation rules by creating several validation rules sets for a single validation. Nonetheless, if several validation rules sets have been created for a single validation, only the latest one will be used by the validation engine during the data transmission once the validation will be published (see paragraph ‘Publishing validations’). To be able to create a validation rules set, the validation has to be locked beforehand (which is already the case if the validation has just been created otherwise see paragraph ‘Locking a validation’). Once locked, the validation has to be opened by clicking on the validation entry (if the DMA did not already do this to lock the validation). The Manage Validation – Data Structure Definition form is then displayed (see Figure 9). It summarizes the global properties of the validation (see Table 6), lists the existing validation rules sets if any (see Table 7) and allows the DMA to perform several actions (see Table 8). Furthermore, the DMA is able to provide a description for the validation in this form. Figure 9: Empty Manage Validation – Data Structure Definition form To create a validation rules set, the DMA has to click on the [Add Rules Set] button. This will launch the Manage Validation – Add a rule set to the DSD form (see Figure 10) where the DMA has to provide the ID and, optionally, the description of the validation rules set (the version is automatically computed) (see Table 9). January 2011 Page 11 Figure 10: Manage Validation – Add a rule set to the DSD form Once done, the DMA has to click on the [Save] button (see Table 10): the new validation rules set is created and is then listed as follows: Figure 11: Manage Validation – DSD form with one validation rules set If required, the validation rules set ID and description can be updated directly from within this form. 3.4 Creating a validation rule Now that a validation rules set has been created, the validation rules can be finally defined. If several validation rules sets exist, the DMA has to first select the desired one and then click on the [Edit Rules Set] button. Otherwise, he has to directly click on this button. The validation rules editor (which is actually a Java applet) then appears as follows, allowing the DMA to manage the validation rules: January 2011 Page 12 Figure 12: Validation rules editor The upper part of the form allows the DMA to select the dimensions of the observation value(s) for which he wants to create a validation rule (see Table 11) while the bottom part lists all the currently defined validations rules (see Table 12) and allows the DMA to perform several actions (see Table 13). The aforementioned observation values dimensions are related to the code lists defined in the DSD relative to the transferred data files. As depicted in the following excerpt of the VALIDATION_SPR_DSD DSD file used throughout this manual (see Figure 13), the dimensions DVSN1, DVSN2 and DVSN3 are respectively linked to code lists VALIDATION_SPR_NOMENCLATURE_1, VALIDATION_SPR_NOMENCLATURE_2 and VALIDATION_SPR_NOMENCLATURE_3. January 2011 Page 13 Figure 13: Excerpt of the DSD used throughout the manual 3.4.1 Defining the scope of a validation rule A validation rule can have different scopes: January 2011 Page 14 • • • it can be specific to a dedicated observation value it can target a range of observation values it can target every observation values To target a specific observation value, the DMA has to provide all the dimensions of the observation value. To target a range of observation values, the DMA has to provide the dimensions that are common to the observation values, leaving the other dimensions blank. To target every observation values, the DMA has to leave all the dimensions blank. To specify a dimension, the DMA has to select a value from the dropdown list under the corresponding dimension as follow: Figure 14: Validation rules editor – Specifying a dimension by label To display the codes (defined in the relative code lists) instead of the labels inside the dropdown, the DMA can click on the [Codes] button (see Table 11). The dropdown will then appear as follow: Figure 15: Validation rules editor – Specifying a dimension by code 3.4.2 Defining a validation rule Two kinds of validation rules can be created: normal rules and advanced ones. January 2011 Page 15 Normal rules are validation rules that do not refer to other observation values. As such, they are also known as intra-record rules. Furthermore, they are applicable to a specific observation value, to a range of observation values or even to every observation value. On the contrary, advanced rules are validation rules that can refer to one or several other observation values. As such, they are also known as inter-record rules. Additionally, they are applicable to a specific observation value only. 3.4.2.1 Normal rules To create a normal rule, the DMA first has to specify the dimensions of the targeted observation value(s). Once done, he has to click on the [Add] button which will display the following dedicated popup window: Figure 16: Normal validation rule popup window From within this popup window, the DMA can: • • • • • • • • activate or deactivate the validation rule specify to which country linked to the dataset the validation rule is dedicated (either all countries or a specific one) specify if the observation value has to be mandatory and, if so, the relative error severity level (see below) specify if the observation value has to be inside a given range and, if so, the relative error severity level specify if the observation value has to fulfil a given expression (see below) and, if so, the relative error severity level specify the error message that will appear in the validation report if the observation value does not fulfil the above mentioned expression specify if the validation rule has to be applied only for observation values whose reference period is greater or equal to a given “valid from” period specify if the validation rule has to be applied only for observation values whose reference period is lower or equal to a given “valid to” period January 2011 Page 16 For the three kinds of validation mentioned above, a specific error severity level can be defined: either Basic or Critical. During the validation process, if an observation value does not fulfil a validation defined as Basic, then a corresponding warning will be listed in the validation report but the data file transfer will not be blocked because of this warning. In the other hand, if an observation value does not fulfil a validation defined as Critical, then a corresponding error will be listed in the validation report and the data file transfer will be blocked because of this error. To define a validation based on an expression, the following keywords can be used: Keyword ABS SQRT NOT OR AND THIS * + / < <= <> = > >= ( ) Description Absolute value Square root Logical not Logical or Logical and Reference to the current observation value Multiplication Addition Subtraction Division Lower than Lower or equal to Not equal Equal Greater than Greater or equal to Opening bracket Closing bracket For instance, the following expression will check that the observation value that is being validated is greater than 100 but is not equal to 150: (THIS > 100) AND (THIS <> 150) 3.4.2.2 Advanced rules To create an advanced rule, the DMA first has to specify each dimension of the targeted observation value. Once done, he has to click on the [Add Advanced Rule] button which will display the following dedicated popup window: January 2011 Page 17 Figure 17: Advanced validation rule popup window As previously mentioned, the aim of advanced rules is to be able to compare a specific observation value to one or several other observation values. The only constraint is that the observation value(s) that is(are) referenced has(have) to have only one dissimilar dimension to the ones of the observation value that is currently validated. For instance, comparing • Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=B, Dim3=C is allowed • Dim1=A, Dim2=B, Dim3=C to Dim1=A, Dim2=Y, Dim3=C is allowed • Dim1=A, Dim2=B, Dim3=C to Dim1=A, Dim2=B, Dim3=Z is allowed while comparing • Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=Y, Dim3=C is not allowed • Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=B, Dim3=Z is not allowed • Dim1=A, Dim2=B, Dim3=C to Dim1=A, Dim2=Y, Dim3=Z is not allowed • Dim1=A, Dim2=B, Dim3=C to Dim1=X, Dim2=Y, Dim3=Z is not allowed To simplify the expression, only the dissimilar dimension of the targeted observation value has to be referenced when comparing observation values. To do so, the syntax used for the reference is based on the grid displaying the dimensions: • • it begins with a word (from A to Z, then AA to ZZ, etc…) which points to the column displaying the dissimilar dimension and it is followed by a number which points to the index of the code in the dropdown list relative to the dimension For instance, as depicted in Figure 18, C2 references the second code (VSN1-LB@CB) of the dimension displayed in column C (DVSN1). January 2011 Page 18 Figure 18: Referencing another observation value As such, if the observation value defined by dimensions DVSN1=VSN1-LC@CC, DVSN2=VSN2-L03@C03, DVSN3=VSN3-LCC@CCC has to be greater than the observation value defined by dimensions DVSN1=VSN1-LB@CB, DVSN2=VSN2-L03@C03, DVSN3=VSN3-LCC@CCC then the following expression has to be used: THIS > C2 The popup depicted in Figure 18 is displayed when clicking on the [View Codes] button from within the Advanced validation rule popup window (see Table 15). Furthermore, the following keywords can also be used in addition to the ones available for normal rules expressions: Keyword AVG MAX MIN SUM Description Average value in list Maximum value in list Minimum value in list Sum of values For instance, the following expression will check that the observation value that is being validated has to be equal to the maximum value of the three observation values for which the code of the dimension displayed in column E varies from the second one to the fourth one: THIS = MAX(E2:E4) January 2011 Page 19 4 Testing a validation rules set Before publishing a validation, the accuracy of the validation rules should be thoroughly tested. By clicking on the [Test rules] button from within the Manage Validation – DSD Form form for the required validation, the DMA can verify the validation rules of the latest validation rules set against a data file of his own for a given country and period. This has no impact on the validation of the transferred data files. To do so, the Manage Validation – test rules form will appear as follows, allowing the DMA to specify the country and the period of the data to validate from a given data file (see Table 18). Figure 19: Manage Validation – Test rules form Once the information has been provided, the DMA has to click on the [Perform] button to launch the validation process (see Table 19). When the validation process has finished, the validation report then appears on the screen has follows: Figure 20: Validation report after testing the validation process January 2011 Page 20 The DMA can then either return to the Manage Validation – DSD Form form by clicking on the [Go the manage validation form] button or return to the Manage Validation – Test rules form by clicking on the [Go to the Test rules form] button. 4.1 Analyzing a validation report The validation reports are divided in two parts: • the Summary part which outlines: o the version of the validation engine that has been used o the date of the validation process o the start and end time of the validation process o the dataset, country and reference period of the provided data o the name of the data file o the number of raised errors o the number of raised warnings • the Validation Log part which outlines: o the key family defined in the data file (e.g. ESTAT:VALIDATION_SPR_DSD:6.0) indicating which DSD to use for the validation process when several DSDs are published for the relevant dataset (see paragraph ‘Publishing validations’) o the DSD used for the validation process o a description for each of the raised errors and warnings January 2011 Page 21 5 Publishing validations In order to activate the validation process for transferred data files, at least one validation has to be published beforehand for the relative dataset. To do so, the DMA has to click on the [Publish] button from within the Manage Validation DSD Form form (see Table 8). Once the validation has been published, the Manage Validation form is displayed and the corresponding entry is marked as active as follows: Figure 21: Manage Validation form with the newly published validation 5.1 Validation mechanism Even though several DSDs can be published for a single dataset, data files will nonetheless be validated against one and only one published validation. If only one validation is published for a given dataset, the transferred data files will then be validated against this validation even if the DSD referenced in the key family defined in the data files does not correspond to the DSD relative to the published validation. When several validations are used for a give dataset, then the validation engine looks at the DSD referenced in the key family defined in the transferred data files. The following cases may occur: • • • • the key family is wrongly defined: an error occurs and the validation process is aborted the key family references a DSD that is not linked to the dataset: the validation process is aborted as no default DSD was found to validate the data file the key family references a DSD that is indeed linked to the dataset but for which the relevant validation is not published: the validation process is also aborted as no default DSD was found to validate the data file the key family references a DSD that is indeed linked to the dataset and for which the relevant validation is indeed published: the data file is validated against the validation relative to the defined DSD January 2011 Page 22 Of course, the validation process will not be launched if no validation is published for a given dataset. If no validation rules set is defined in the validation to be processed, or if no validation rule is defined in the latest validation rules set, only the structure of the transferred data files will be validated. If several validation rules sets are defined in the validation to be processed, then the validation rules of the latest validation rules set will be used by the validation engine. The validation rules will be applied according to their (optional) validity period and to the reference period of the observation values to validate: • • if no validity period is defined then the validation rule will always be applied if a validity period is defined then the validation rule will be applied to observation values whose reference period fulfil the defined validity period January 2011 Page 23 6 Managing validations As previously seen, when clicking on a validation entry in the Manage Validation form, the DMA can either edit the corresponding validation details or view them in read-only mode. Additionally, the actions described afterwards are also available. 6.1 Locking a validation To be able to manage validations, they need most of the time to be locked beforehand. One or several validations can be locked at a time through the Manage Validation form (see Figure 8). To lock a single validation, the DMA has two possibilities: • either by selecting the corresponding checkbox and by clicking on the [Lock] button • or by directly clicking on the validation entry and by clicking on the [OK] button on the following confirmation message: Figure 22: Confirmation message when clicking on a validation entry If the validation was previously published, the following confirmation message would appear instead, forcing the validation to be also unpublished: Figure 23: Confirmation message when clicking on a published validation entry To lock several validations at once, the DMA has to select the corresponding checkboxes and to click on the [Lock] button. If the DMA clicks on the [Lock] button while no validation was selected, the following error message appears: January 2011 Page 24 Figure 24: Error message when locking a validation If the DMA tries to lock an already locked validation, the following error message appears: Figure 25: Error message when locking an already locked validation Otherwise, if the locking was successfully made, the validation then appears with a red lock icon as follows: Figure 26: Validation listed as locked 6.2 Unlocking a validation Once the DMA has finished working on a validation, he has to unlock it in order for others DMAs to be able to work on it. One or several validations can be unlocked at a time through the Manage Validation form (see Figure 8). To unlock a validation, the DMA has to select the corresponding checkbox and to click on the [Unlock] button. If the DMA clicks on the [Unlock] button while no validation was selected, the following error message appears: January 2011 Page 25 Figure 27: Error message when unlocking a validation If the DMA tries to unlock an already unlocked validation, the following error message appears: Figure 28: Error message when locking an already locked validation Otherwise, if the unlocking was successfully made, the validation then appears without the red lock icon anymore as follows: Figure 29: Validation listed as not locked 6.3 Deleting a validation The DMA can delete validations through the Manage Validation form (see Figure 8) providing that they are locked and inactive. To do so, he has to select one or several locked and inactive validations and to click on the [Delete] button. The following confirmation message then appears, allowing the DMA to confirm or abort the deletion: January 2011 Page 26 Figure 30: Confirmation message when deleting validations Should the validations be not locked or/and still published, the following error message would appear: Figure 31: Error message when deleting validations Otherwise, the validation(s) is(are) deleted and the following confirmation message listing the deleted validation(s) appears: Figure 32: Confirmation message when deleting validations January 2011 Page 27 7 Managing validation rules sets The following actions are available through the Manage Validation – DSD Form form. 7.1 Copying a validation rules set To duplicate a set of validation rules, the DMA can copy a validation rules set through the Manage Validation – DSD Form form (see Figure 11). This is nonetheless possible only if the parent validation is locked and inactive. To do so, he has either to select the validation rules set and to click on the [Copy Rules Set] button if several validation rules sets exist or to directly click on the [Copy Rules Set] button. This will launch the Manage Validation – Create a copy of the rules set form (see Figure 33) where the DMA has to provide the ID and, optionally, the description of the validation rules set (the version is automatically computed) (see Table 16). Figure 33: Manage Validation – Add a rule set to the DSD form Once done, the DMA has to click on the [Save] button (see Table 17): the new validation rules set is created and is then listed as the latest entry in the Manage Validation –DSD Form form. 7.2 Deleting a validation rules set The DMA can delete validation rules sets through the Manage Validation – DSD Form form (see Figure 11) providing that the parent validation is locked and inactive. To do so, he has either to select one or several validation rules sets and to click on the [Delete Rules Set] button if several validation rules sets exist or to directly click on the [Delete Rules Set] button. The following confirmation message then appears, allowing the DMA to confirm or abort the deletion: Figure 34: Confirmation message when deleting validation rules sets January 2011 Page 28 The validation(s) is(are) deleted and the following confirmation message listing the deleted validation rules set(s) appears: Figure 35: Confirmation message when deleting validation rules sets January 2011 Page 29 8 Managing validation rules The following actions are available through the validation rules editor (see Figure 12). 8.1 Editing a validation rule If a validation rule needs to be modified, the DMA has to select it and to click on the [Edit] button. According to the type of the selected validation rule, the appropriate popup window will then appear. 8.2 Removing a validation rule If a validation rule needs to be deleted, the DMA has to select it and to click on the [Remove] button. January 2011 Page 30 9 Validating data files The information contained in this section is a quick overview of how the validations are processed when transferring data files to EDAMIS. Senders from Member States have several possibilities to send data files to EDAMIS: either by using EWP or by using EWA. Both transmission modes will be explained here. 9.1 Sending data files through EWP To transfer data files to EDAMIS through EWP, end-users have to have SENDER privileges. Once connected to EWP, they have to go to the Send Datafile interface by selecting the Transmission >> Send Datafile menu entry as follows: Figure 36: Send Datafile menu entry In EWP, the Send Datafile interface allows two modes for the upload of data files: • • the Simple upload form the Advance and highly secure upload form The advantage of the advanced upload form is that it allows the encryption of the data. To do so, it relies on a Java applet which is also the drawback of this transmission mode. Whatever the chosen mode, when selecting a data file (and specifying the dataset if the data file does not respect the dataset naming convention), the Validation? checkbox is ticked if ever a published validation exist for the targeted dataset (see Figure 37). January 2011 Page 31 Figure 37: Send Datafile form when selecting a data file to be validated 9.1.1 Sending data files containing critical errors If the SENDER sends a data file that contains at least one critical error, the validation process raises an error and the data file transfer is aborted as depicted below: Figure 38: Uploading a data file containing a (critical) error (Advanced upload form) Figure 39: Uploading a data file containing a (critical) error (Simple upload form) The data file is then listed as a non transmitted file with validation status set to Validation NOT OK [-2] as follows: January 2011 Page 32 Figure 40: Send Datafile form after data files upload Nonetheless, the SENDER has the possibility to force the sending of the data file by clicking on the corresponding [force delivery] button. By clicking on the corresponding [view log files] button, a report such as the following one will be displayed: Figure 41: Validation report with a (critical) error 9.1.2 Sending data files containing basic errors If the SENDER sends a data file containing a basic error, the validation process raises a warning but the data file is transferred nonetheless as depicted below (unless it contains at least a critical error): Figure 42: Uploading a data file containing a warning (basic error) (Advanced upload form) Figure 43: Uploading a data file containing a warning (basic error) (Simple upload form) January 2011 Page 33 The data file is then listed as a transmitted file with validation code set to Validation with warnings [-1] (see Figure 40). By clicking on the corresponding [view log files] button, a report such as the following one will be displayed: Figure 44: Validation report with a warning (basic error) 9.1.3 Sending correct data files If the SENDER sends a data file with no error of any kind, the validation process ends successfully as depicted below: Figure 45: Uploading a data file with no error (Advanced upload form) Figure 46: Uploading a data file with no error (Simple upload form) The data file is then listed as a transmitted file with validation code set to Validation successful [0] (see Figure 40). By clicking on the corresponding [view log files] button, a report such as the following one will then be displayed: January 2011 Page 34 Figure 47: Validation report with no error 9.1.4 Validation reports availability Once the SENDER has logged out from EWP, the validation reports are not available anymore through the Send Datafile form. Nonetheless, validation reports are also sent by mail to the SENDER and to the RECEIVER at the end of the validation process. 9.2 Sending data files through EWA To transfer data files to EDAMIS through EWA, end-users have to have SENDER privileges. Once connected to EWA, they have to go to the Send Datafiles interface by selecting the Send Datafile >> Send Datafile menu entry as follows: Figure 48: EWA - Send Datafile menu entry Contrary to EWP, when selecting a data file on EWA(and specifying the dataset if the data file does not respect the dataset naming convention) the Valid? checkbox is unticked (see Figure 49). Figure 49: EWA - Send Datafiles form when selecting a data file to be validated January 2011 Page 35 As of today (EWA 2.9.9b), validations are not performed locally but on the server once the data files have been transmitted to EWP. Contrary to EWP, validation reports are not available through EWA but they are sent by mail to the SENDER and to the RECEIVER at the end of the validation process. Figure 50: EWA - Send Datafiles form after data files upload January 2011 Page 36 10 Annex Manage Validation form – Selection area The selection area allows the DMA to shorten the list of existing validations by means of filters on theme, domain and dataset. 1 Theme This filter allows the DMA to select the theme. 2 Domain This filter allows the DMA to select the domain. 3 Dataset This filter allows the DMA to select the dataset. 4 [View] button Once the filters have been changed, the user has to click on the [View] button to refresh the page accordingly. Table 1: Manage Validation form – Selection area Manage Validation form – Validations description area This area lists the existing validations for the given filters. If the DMA clicks on one of them, the corresponding validation is opened either in read only mode (if the validation was not locked by the DMA) or in edit mode. 1 Checkbox These checkboxes enable the selection of one or several validations on which an operation will be performed. 2 Dataset This indicates the ID of the dataset to which the validation is attached. 3 Label This indicates the label of the dataset relative to the corresponding validation. 4 Dsd ID This indicates the ID of the DSD relative to the corresponding validation. 5 Dsd Version This indicates the version of the DSD relative to the corresponding validation. 6 Dsd Agency This indicates the agency of the DSD relative to the corresponding validation. January 2011 Page 37 7 Dsd Source This indicates the source of the DSD relative to the corresponding validation. 8 Lock This indicates if the corresponding validation is locked (marked with a red lock icon) or not. 9 Creation This indicates when and by whom the corresponding validation has been created. 10 Modification This indicates when and by whom the corresponding validation has been lately modified. 11 Active This indicates if the corresponding validation is published or not. Table 2: Manage Validation form – Validations description area Manage Validation form – Action buttons This allows the DMA to perform selected actions on validations. 1 [Lock] button (visible only when entries are listed) By selecting an entry and by clicking on the [Lock] button, the DMA can lock the corresponding validation to make it editable. 2 [Unlock] button (visible only when entries are listed) By selecting an entry and by clicking on the [Unlock] button, the DMA can unlock the corresponding validation to make it available for edition to other users. 3 [Associate a dsd] button By clicking on the [Associate a dsd] button, the DMA can create a new validation through the Manage Validation – Associate a DSD form. 4 [Delete] button (visible only when entries are listed) By selecting one or several entries and by clicking on the [Delete] button, the DMA can delete the corresponding validation(s) providing that it(they) is(are) locked and inactive. Table 3: Manage Validation form – Action buttons Manage Validation - Associate a DSD form – Definition area The definition area allows the DMA to specify the properties of the DSD to be associated to the current dataset. 1 DSD ID January 2011 Page 38 This allows the DMA to provide the ID of the DSD to be associated to the current dataset. 2 DSD Version This allows the DMA to provide the version of the DSD to be associated to the current dataset. 3 DSD Agency This allows the DMA to provide the agency of the DSD to be associated to the current dataset. Table 4: Manage Validation - Associate a DSD form – Definition area Manage Validation - Associate a DSD form – Action buttons This allows the DMA to finalize or to abort the association process. 1 [Associate a dsd] button By clicking on the [Associate a dsd] button, the DMA is able to associate to the current dataset the DSD defined by the properties provided beforehand. A warning message will be displayed if the DSD cannot be retrieved from the SDMX Registry or if the DSD has been already associated to the current dataset. 2 [Cancel] button The [Cancel] button allows the DMA to abort the association process. Table 5: Manage Validation - Associate a DSD form – Action buttons Manage Validation – DSD Form form – Summary area This area summarizes the global properties of the current validation. 1 Dataset (read only) This indicates the ID of the dataset relative to the current validation. 2 DSD Id (read only) This indicates the ID of the DSD relative to the current validation. 3 Version (read only) This indicates the version of the DSD relative to the current validation. 4 Agency (read only) This indicates the agency of the DSD relative to the current validation. 5 Description This allows the DMA to provide a description for the current validation. January 2011 Page 39 6 (read only) Creation This indicates when and by whom the current validation has been created. 7 (read only) Modification This indicates when and by whom the current validation has been lately modified. 8 (read only) Active This indicates if the current validation is published or not. Table 6: Manage Validation – DSD Form form – Summary area Manage Validation – DSD Form form – Validation rules sets description area This area lists the existing validation rules sets for the current validation. 1 Checkbox (visible only when several entries are listed) These checkboxes enable the selection of one or several validation rules sets on which an operation will be performed. 2 Rules Set Name This indicates the ID of the corresponding validation rules set. If the DMA needs to modify it, this is the only place to do so. A confirmation message will appear if the DMA clicks on an action button. 3 Description This indicates the description of the corresponding validation rules set. If the DMA needs to modify it, this is the only place to do so. A confirmation message will appear if the DMA clicks on an action button. 4 (read only) Version This indicates the (automatically computed) version of the corresponding validation rules set. Table 7: Manage Validation – DSD Form form – Validation rules sets description area Manage Validation – DSD Form form – Action buttons This allows the DMA to perform selected actions on validation rules sets. When only one validation rules set is defined, no checkbox is displayed and the DMA can directly click on the appropriate action button. 1 [Add Rules Set] button (enabled only when the validation is locked) By clicking on the [Add Rules Set] button, the DMA can create a new validation rules set through the Manage Validation – Add a rule set to the DSD form. 2 [Copy Rules Set] button January 2011 (visible only when entries are listed Page 40 and enabled only when the validation is locked) By selecting an entry and by clicking on the [Copy Rules Set] button, the DMA can duplicate the selected validation rules set. 3 [Edit Rules Set / View Rules Set] button (enabled only when entries are listed) By selecting an entry and by clicking on the [Edit Rules Set] button, the DMA can edit the validation rules of the selected validation rules set through the validation rules editor. If the validation was not locked beforehand by the DMA, the button caption is set to [View Rules Set] and the validation rules editor is opened in read only mode. 4 [Delete Rules Set] button (enabled only when entries are listed and when the validation is locked) By selecting an entry and by clicking on the [Delete Rules Set] button, the DMA can delete the corresponding validation rules set. 5 [Test rules] button (visible only when entries are listed) By selecting an entry and by clicking on the [Test rules] button, the DMA can test the validation rules of the corresponding validation rules set against a SDMX-ML data file through a dedicated form. 6 [Save] button By clicking on the [Save] button, the DMA can save the current validation. 7 [Publish / Unpublish] button By clicking on the [Publish] button, the DMA can publish the current validation. By clicking on the [Unpublish] button, the DMA can unpublish the current validation. 8 [Cancel] button By clicking on the [Cancel] button, the DMA can leave the Manage Validation – DSD Form form. Table 8: Manage Validation – DSD Form form – Action buttons Manage Validation – Add a rules set to the DSD form – Definition area The definition area allows the DMA to specify the properties of the validation rules set to be added to the current validation. 1 Rules set Id (mandatory) This allows the DMA to provide the ID of the validation rules set to be added to the current validation. 2 Description (optional) This allows the DMA to provide the description of the validation rules set to be added to the current validation. January 2011 Page 41 3 (read only) Version This automatically computed value indicates the sort order of the validation rules set to be added to the current validation. Table 9: Manage Validation - Add a rules set to the DSD form – Definition area Manage Validation - Add a rules set to the DSD form – Action buttons This allows the DMA to finalize or to abort the creation process. 1 [Save] button By clicking on the [Save] button, the DMA is able to associate to the current validation the validation rules set defined by the properties provided beforehand. 2 [Cancel] button The [Cancel] button allows the DMA to abort the creation process. Table 10: Manage Validation - Add a rules set to the DSD form – Action buttons Manage Validation – Validation rules editor – Selection area The selection area allows the DMA to select the dimensions of the observation value(s) for which he wants to create a validation rule. As a reminder, it also indicates the current dataset, the current validation rules set ID and version as well as the current DSD ID. 1 Dimensions By clicking beneath a dimension cell, a dropdown list appears allowing the DMA to select the desired value for the corresponding dimension. By default, the dropdown list displays the labels of the dimension items. 2 [Codes / Labels] button By clicking on the [Codes] button, the aforementioned dropdown lists display the codes of the dimensions items instead of their labels. The button caption then toggles to [Labels] in order to switch back to the display of labels. Table 11: Validation rules editor – Selection area Manage Validation – Validation rules editor – Validation rules description area This area lists the existing validation rules for the current validation rules set. 1 Dimensions Names This indicates the dimensions of the observation value(s) targeted by the corresponding validation rule. January 2011 Page 42 2 Type This indicates the type of the corresponding validation rule: normal or advanced. 3 Country This indicates the country targeted by the corresponding validation rule. 4 Mandatory If checked, this indicates that the targeted observation value(s) has(have) to be mandatory. Furthermore, the relative severity level (basic/critical) is also indicated. 5 Range of values If provided, this indicates that the targeted observation value(s) has(have) to be in between the specified values. Furthermore, the relative severity level (basic/critical) is also indicated. 6 Active If checked, this indicates that the targeted observation value(s) has(have) to be validated against the corresponding validation rule. 7 Expression If provided, this indicates that the targeted observation value(s) has(have) to fulfil the specified expression. Furthermore, the relative severity level (basic/critical) is also indicated. 8 Valid If provided, this indicates that the corresponding validation rule will be applied to the targeted observation value(s) whose reference period is greater or equal to the specified period. 9 Valid To If provided, this indicates that the targeted observation value(s) has(have) to be validated against the corresponding validation rule only if its(their) reference period is lower or equal to the specified period. Table 12: Validation rules editor – Validation rules description area Manage Validation – Validation rules editor – Action buttons This allows the DMA to perform selected actions on validation rules. 1 [Add] button By optionally specifying one or several dimensions and by clicking on the [Add] button, the DMA can create a regular rule through a dedicated popup window. 2 [Add Advanced Rule] button By specifying all required dimensions and by clicking on the [Add Advanced Rule] button, the DMA can create an advanced rule through a dedicated popup window. January 2011 Page 43 3 [Remove] button By selecting a rule and by clicking on the [Remove] button, the DMA can remove the corresponding validation rule. 4 [Edit] button By selecting a rule and by clicking on the Edit button, the DMA can edit the corresponding validation rule through the corresponding popup window. 5 [Save] button By clicking on the Save button, the DMA can save the validation rules of the current validation rules set. 6 [Exit] button By clicking on the Exit button, the DMA can exit the validation rules editor and return to the Manage Validation – DSD Form form. Table 13: Validation rules editor – Action buttons Validation rules editor – Normal Rule popup window This popup window allows the DMA to create or edit a normal validation rule, aka intra-record rule. 1 Activate This allows the DMA to specify if the current validation rule has to be applied during the validation process. 2 Country This allows the DMA to specify to which country linked to the dataset the current validation rule is dedicated (either all countries or a specific one). 3 Mandatory This allows the DMA to specify if the observation value has to be mandatory and, if so, the relative error severity level if the observation value is not provided. 4 Range This allows the DMA to specify if the observation value has to be inside a given range and, if so, the relative error severity level if the observation value is outside this range. 5 Expression This allows the DMA to specify if the observation value has to fulfil a given expression and, if so, the relative error severity level if the observation value does not fulfil this expression. 6 Error message This allows the DMA to specify the error message that will appear in the validation report if the observation value does not fulfil the above mentioned expression. January 2011 Page 44 7 Valid From Period This allows the DMA to specify if the current validation rule has to be applied only for observation values whose reference period is greater or equal to a given “valid from” period. 8 Valid To Period This allows the DMA to specify if the current validation rule has to be applied only for observation values whose reference period is lower or equal to a given “valid to” period. 9 [OK] button By clicking on the [OK] button, the DMA creates or updates the current validation rule. 10 [Cancel] button By clicking on the [Cancel] button, the DMA can abort the creation or update of the current validation rule. Table 14: Validation rules editor – Normal Rule popup window Validation rules editor – Advanced Rule popup window This popup window allows the DMA to create or edit an advanced validation rule, aka inter-record rule. 1 Activate This allows the DMA to specify if the current validation rule has to be applied during the validation process. 2 Country This allows the DMA to specify to which country linked to the dataset the current validation rule is dedicated (either all countries or a specific one). 3 Mandatory This allows the DMA to specify if the observation value has to be mandatory and, if so, the relative error severity level if the observation value is not provided. 4 Expression This allows the DMA to specify if the observation value has to fulfil a given expression and, if so, the relative error severity level if the observation value does not fulfil this expression. By clicking on the [View Codes] button, a popup window displaying the available values for each dimension is displayed in order to allow the DMA to retrieve the correct combination to use as reference in the expression (see paragraph ‘Advanced rules’). 5 Error message This allows the DMA to specify the error message that will appear in the validation January 2011 Page 45 report if the observation value does not fulfil the above mentioned expression. 6 Valid From Period This allows the DMA to specify if the current validation rule has to be applied only for observation values whose reference period is greater or equal to a given “valid from” period. 7 Valid To Period This allows the DMA to specify if the current validation rule has to be applied only for observation values whose reference period is lower or equal to a given “valid to” period. 8 [OK] button By clicking on the [OK] button, the DMA creates or updates the current validation rule. 9 [Cancel] button By clicking on the [Cancel] button, the DMA can abort the creation or update of the current validation rule. Table 15: Validation rules editor – Advanced Rule popup window Manage Validation – Create a copy of the rules set form – Definition area The definition area allows the DMA to specify the properties of the validation rules set duplicate. 1 Rules set Id (mandatory) This allows the DMA to provide the ID of the validation rules set duplicate. 2 Description (optional) This allows the DMA to provide the description of the validation rules set duplicate. 3 Version (read only) This automatically computed value indicates the sort order of the validation rules set duplicate. Table 16: Manage Validation - Create a copy of the rules set form – Definition area Manage Validation - Create a copy of the rules set form – Action buttons This allows the DMA to finalize or to abort the duplication process. 1 [Save] button By clicking on the [Save] button, the DMA launches the duplication process. 2 [Cancel] button The [Cancel] button allows the DMA to abort the duplication process. January 2011 Page 46 Table 17: Manage Validation - Create a copy of the rules set form – Action buttons Manage Validation – Test rules form – Definition area The definition area allows the DMA to specify the properties of the data file to test the validation rules against. 1 Dataset (read only) This indicates as a reminder the parent dataset of the current validation. 2 Country (mandatory) This allows the DMA to specify for which country the data have to be tested. 3 Year - Period (mandatory) This allows the DMA to specify for which reference period the data have to be tested. 4 Data File (mandatory) This allows the DMA to specify the data file containing the data to be tested. Table 18: Manage Validation - Test rules form – Definition area Manage Validation - Test rules form – Action buttons This allows the DMA to finalize or to abort the test of the validation process. 1 [Perform] button By clicking on the [Perform] button, the DMA launches the test of the validation process. 2 [Cancel] button The [Cancel] button allows the DMA to abort the test of the validation process. Table 19: Manage Validation - Test rules form – Action buttons January 2011 Page 47 11 Glossary Concept Definition CIRCA CIRCA is an extranet tool developed by the European Commission and tuned towards Public Administrations’ needs. It enables a given community geographically spread across Europe to form an online ‘Interest Group’, thereby providing a private space on the Internet where members can share information and documents, participate in discussion forums and make use of various other functionalities. URL: http://circa.europa.eu/ CSV Comma-Separated Values file format. Data file Physical data file transmitted which should normally correspond to an instance of a dataset (a physical instance of a dataset occurrence). Dataset Dataset structure with a specific periodicity and to which is usually linked a deadline for transmission. The data content of a dataset is homogeneous. Dataset naming Convention used by all production units at Eurostat for uniformly naming a dataset. It composed of the domain name, the statistical convention table name and the periodicity, all separated by an underscore. The dataset naming convention is recognised by all third-party applications at Eurostat. Dataset occurrence Occurrence of a dataset for one country and one period or time series or sequence. For Web Forms it is called a ‘Web Form instance’. Dataset structure Concept of one single data structure associated to one or several statistical tables (closely linked into the same statistical domain). Domain Consistent group of datasets that are closely linked together (belong to the same statistical area and based on the same legal act(s) or voluntary agreements). EDAMIS Electronic Data files Administration and Management Information System. EWA EDAMIS Web Application. EWP EDAMIS Web Portal. SDMX-ML XML format for the exchange of SDMX-structured data and metadata. The SDMX format (Statistical Data and Metadata Exchange) is the result of cooperation between the BIS, the ECB, Eurostat, the IMF, the OECD and the UN to explore common e-standards and ongoing standardisation activities that could afford more efficiency in the field of statistical information. Theme Theme of the Statistical Programme. January 2011 Page 48 12 History of Changes 31.01.2011 EDAMIS release 3.1c Document version 1.0 Status draft How to set up and use the validation in EDAMIS January 2011 Page 49
© Copyright 2024