07 XML based technologies used in SDMX@Observatorio Ocupacional del SINAFOCAL th

The student book is the sixth one of a set of student books which one of a set of student books which altogether altogether provides the complete set of information provides the complete set of information to master to master SDMX student book six helps you to SDMX student book six helps you to understand understand the XML technology the different ways in the XML technology the different ways in which which XML is used at Eurostat are presented XML is used at Eurostat are presented here here and the different applications that have and the different applications that have already been created already been created are introduced detailed step-by-step are introduced detailed step-by-step exercises are provided exercises are provided explaining the different specific explaining the different specific aspects of SDMX aspects of SDMX and SDMX compliant applications and SDMX compliant applications this student book aims at providing an this student book aims at providing an explanation explanation about the XML technologies that can be about the XML technologies that can be applied applied in the use of SDMX we will see in the use of SDMX we will see one chapter explaining the basis of XML one chapter explaining the basis of XML and schema and name spaces and schema and name spaces another chapter with the explanation of another chapter with the explanation of what what xslt is and real examples xslt is and real examples of conversions between SDMX files of conversions between SDMX files and then an introduction of web services and then an introduction of web services to give us a better understanding of how to give us a better understanding of how to apply them to apply them to SDMX standards no to SDMX standards no specific prerequisite is needed to specific prerequisite is needed to understand understand the content extensible markup language the content extensible markup language XML is a markup language used to XML is a markup language used to describe the content and structure of describe the content and structure of data in a document data in a document XML is an industry standard for XML is an industry standard for delivering delivering content on the Internet because IT content on the Internet because IT provides a facility provides a facility to define new tags XML is also to define new tags XML is also extensible like HTML extensible like HTML XML uses tags to describe content XML uses tags to describe content however rather than focusing on the however rather than focusing on the presentation of content presentation of content the tags in XML describe the meaning the tags in XML describe the meaning and hierarchical structure of data and hierarchical structure of data the XML syntax uses matching the XML syntax uses matching start and end tags to markup information start and end tags to markup information information delimited by tags is called information delimited by tags is called an element every XML document has a an element every XML document has a single root element single root element which is the top level element that which is the top level element that contains all the other elements contains all the other elements XML documents begin with the processing XML documents begin with the processing instruction instruction while IT is not required its presence while IT is not required its presence explicitly identifies the document as an explicitly identifies the document as an XML document XML document and indicates the version of XML to and indicates the version of XML to which IT was which IT was authored empty elements authored empty elements have a modified syntax while most have a modified syntax while most elements in a document are elements in a document are wrappers around some content empty wrappers around some content empty elements are simply elements are simply markers where something occurs markers where something occurs the trailing slash in the modified the trailing slash in the modified syntax syntax indicates to a program processing the indicates to a program processing the XML document XML document that the element is empty and no that the element is empty and no matching and matching and tag should be sought XML documents consist of markup and content six kinds of markup and content six kinds of markup can occur in an XML document can occur in an XML document elements entity references elements entity references comments processing instructions comments processing instructions marked sections and marked sections and document type declaration document type declaration XML name spaces are used for providing XML name spaces are used for providing uniquely named elements and attributes uniquely named elements and attributes in order to avoid element name conflicts in order to avoid element name conflicts inside the XML file inside the XML file this ensures re usability in other words this ensures re usability in other words the reuse of elements and attributes the reuse of elements and attributes IT ensures molecularity or the use IT ensures molecularity or the use of elements and attributes from other of elements and attributes from other standard results standard results and extensibility that is to say the and extensibility that is to say the incorporation of elements and attributes incorporation of elements and attributes from from other vocabularies other vocabularies an XML name space is a sequence of an XML name space is a sequence of characters characters identified by a uri reference identified by a uri reference which are used in XML documents as which are used in XML documents as element element types and attribute names types and attribute names in an XML document the uri in an XML document the uri is associated with a prefix and this is associated with a prefix and this prefix prefix is used with each element to indicate is used with each element to indicate to which name space the element belongs to which name space the element belongs to The name space is declared by using a family of reserved attributes family of reserved attributes such an attributes name must either be such an attributes name must either be xmlns xmlns or contain xmlns as a prefix or contain xmlns as a prefix these attributes may be provided either these attributes may be provided either explicitly explicitly or by default an attributes value or by default an attributes value is a uri reference the string is a uri reference the string identifying the name space the name space identifying the name space the name space name name has the characteristic of being unique has the characteristic of being unique here an example of using xmlns here an example of using xmlns as an attribute name We have the value for the name and the value for the attribute and finally value for the attribute and finally the name space declaration We can see another example using xmlns as prefix for another attribute name as prefix for another attribute name if this attribute name is xs if this attribute name is xs and with the following attributes value and with the following attributes value the name space declaration the name space declaration should be as follows A name space declaration applies to all elements elements and attributes names whose prefix and attributes names whose prefix matches the one specified matches the one specified in the declaration of the name space in the declaration of the name space the following is an example of a DSD the following is an example of a DSD where the elements match the prefix where the elements match the prefix structure declared structure declared this DSD belongs to the demography this DSD belongs to the demography domain domain the prefix structure is defined here the prefix structure is defined here we can see the different elements that we can see the different elements that are using the prefix are using the prefix code list and code list and name for example name for example an XML schema defines the permitted an XML schema defines the permitted building blocks building blocks of an XML document using the terms of an XML document using the terms of a schema definition language which is of a schema definition language which is itself written in XML SDMX itself written in XML SDMX makes use of the schema definition makes use of the schema definition language known language known as XML schema published as XML schema published as a w3c recommendation in may 2001 as a w3c recommendation in may 2001 the benefits of using xsd schema are the benefits of using xsd schema are xsd schema is an XML document xsd schema is an XML document xsd schema supports inheritance which xsd schema supports inheritance which means that one schema can means that one schema can inherit from another schema xsd schema inherit from another schema xsd schema provides the ability to define new data provides the ability to define new data types types from the existing data type and to from the existing data type and to specify data types for both specify data types for both elements and attributes the root element elements and attributes the root element of an xsd schema already defined of an xsd schema already defined as an XML document is schema as an XML document is schema the xsd name space declaration is the xsd name space declaration is provided inside provided inside the schema element the name space the schema element the name space declaration defines the XML file declaration defines the XML file as being an xsd schema as being an xsd schema element is the most important part of element is the most important part of the schema the schema the kind of information that must be the kind of information that must be passed from the XML document passed from the XML document is specified at this point is specified at this point element is limited by its type element is limited by its type for example a simple type element for example a simple type element can contain text only whereas an element can contain text only whereas an element of complex of complex type can contain child elements type can contain child elements and attributes the diagram and attributes the diagram provides an overview of the possible provides an overview of the possible types types of XML schema elements One simple type can be atomic or non-atomic if IT or non-atomic if IT is user derived and in the case of being is user derived and in the case of being built built in the possible values are in the possible values are primitive or derived primitive or derived in the case of a complex type IT can be in the case of a complex type IT can be empty simple content or empty simple content or complex content complex content only in the case of complex content only in the case of complex content IT can be a sequence all IT can be a sequence all or choice or choice for example inside one schema created to for example inside one schema created to validate one SDMX data file validate one SDMX data file there are several specifications there are several specifications in this case there is the specification in this case there is the specification of the type postal code type of the type postal code type with one restriction that means with one restriction that means that if there is any element declared of that if there is any element declared of this type this type the value of the data file must be a the value of the data file must be a string string in order to validate one XML file in order to validate one XML file against a schema against a schema the schema location attribute is used the schema location attribute is used inside the root element of the XML file inside the root element of the XML file this attribute consists in nodes this attribute consists in nodes of two components name space of two components name space and schema to reuse and schema to reuse another schema the first step another schema the first step is to include IT the include statement is to include IT the include statement in the original schema must point to the in the original schema must point to the secondary schema secondary schema and the target name space should be the and the target name space should be the name name as on the current working location as on the current working location more than one schema can be included more than one schema can be included by adding several schema declaration by adding several schema declaration inside the include statement inside the include statement the recommended SDMX schema are the recommended SDMX schema are available available through www.sdmx.org through www.sdmx.org with a description of each one according with a description of each one according to the SDMX to the SDMX file IT has to validate file IT has to validate all SDMX ml document types will share all SDMX ml document types will share a common envelope at a message level a common envelope at a message level as well as a set of common low level as well as a set of common low level components components so that header information and basic so that header information and basic structure structure will always be the same here will always be the same here we can find the list of schema used in we can find the list of schema used in SDMX SDMX standard depending of the format standard depending of the format of the message schema for describing all of the message schema for describing all types of structural metadata types of structural metadata for data sets for metadata sets for data sets for metadata sets for related groups of metadata and data for related groups of metadata and data structures structures and for all types of structural objects and for all types of structural objects involved in registry based exchanges involved in registry based exchanges generic data schema for data sharing generic data schema for data sharing exchange in generic format exchange in generic format generic query schema for invoking generic query schema for invoking web services key web services key family specific schema compact format family specific schema compact format key family specific schema for utility key family specific schema for utility format format key family specific schema for key family specific schema for cross-sectional format cross-sectional format generic schema for registry interfaces generic schema for registry interfaces generic schema for reference metadata generic schema for reference metadata sets sets and metadata structure definition and metadata structure definition specific schema specific schema for metadata sets In addition to these different schema standard mappings and corresponding standard mappings and corresponding transformation tools have been developed transformation tools have been developed to create key family schema to to create key family schema to transform transform XML data instances from one XML XML data instances from one XML data description format to another and data description format to another and from these formats from these formats into the corresponding SDMX ml messages into the corresponding SDMX ml messages IT is fact that all formats share a IT is fact that all formats share a common information model common information model that enables this approach to meet the that enables this approach to meet the wide set wide set of SDMX requirements some sdm XML of SDMX requirements some sdm XML schema schema are the same for all key families and are the same for all key families and metadata structure definitions metadata structure definitions these include SDMX message these include SDMX message dot xsd for generically describing the dot xsd for generically describing the basic message basic message structure common to all sdm XML messages Sdmx structure.xsd for describing key families metadata for describing key families metadata structure definitions structure definitions data flows metadata flows data flows metadata flows code lists concepts structure sets code lists concepts structure sets processes hierarchical code lists processes hierarchical code lists and reporting taxonomies and reporting taxonomies SDMX generic data dot SDMX generic data dot xsd for describing data across xsd for describing data across key families for generic processing SDMX query dot xsd for marking up queries against for marking up queries against conformat databases and web services conformat databases and web services SDMX common.Xsd SDMX common.Xsd describing the common constructs used in describing the common constructs used in other schemas Sdmx generic metadata.xsd for generically reporting reference

for generically reporting reference metadata metadata and SDMX registry dot and SDMX registry dot xsd for all interactions xsd for all interactions with the SDMX registry services Note that for all key family specific and metadata structure definition and metadata structure definition specific specific schemas the namespaces must be schemas the namespaces must be constructed constructed following the rules for registry urn following the rules for registry urn identifiers with the addition of a identifiers with the addition of a single field single field at the end of the urn for utility at the end of the urn for utility schemas schemas utility header of SDMX utility header of SDMX utility data dot xsd utility data dot xsd for compact schemas compact for compact schemas compact header of SDMX compact header of SDMX compact data dot xsd data dot xsd for cross-sectional schemas cross for cross-sectional schemas cross header of SDMX cross-sectional data header of SDMX cross-sectional data dot xsd for metadata report schemas dot xsd for metadata report schemas meta data report header of meta data report header of metadata report dot xsd Here we can find some explanations of the meaning of each of the basic the meaning of each of the basic components of the SDMX files components of the SDMX files for example inside one compact file for example inside one compact file where we can find the root compact data where we can find the root compact data element there are also other elements to element there are also other elements to be taken into consideration be taken into consideration there are four different namespaces there are four different namespaces declared declared which will be used inside the rest of which will be used inside the rest of the file the file the basic one the one corresponding to the basic one the one corresponding to the domain the domain of the data file and DSD the one related of the data file and DSD the one related to the compact format to the compact format and the one related to the schema and the one related to the schema instance instance once all the namespaces are described once all the namespaces are described to declare the schemas against which the to declare the schemas against which the compact file will be validated compact file will be validated the xsi schema the xsi schema location element is used location element is used IT is composed by nodes of two elements IT is composed by nodes of two elements the first node the namespace the first node the namespace and the schema location the second node and the schema location the second node the namespace here is a urn and IT the namespace here is a urn and IT is a unique namespace declaration for is a unique namespace declaration for this file this file and finally the last node and finally the last node with the declaration of the namespace with the declaration of the namespace for compact format for compact format and a relative path to the schema and a relative path to the schema this is another example of a compact this is another example of a compact schema created from a previous DSD schema created from a previous DSD used to validate any SDMX data file used to validate any SDMX data file in compact format here in compact format here we can see the main elements of the root we can see the main elements of the root element element there are five namespaces specified in there are five namespaces specified in this case this case the basic namespace for all the schemas the basic namespace for all the schemas the namespace common which is relative the namespace common which is relative to any SDMX schema to any SDMX schema the use of compact namespace because IT the use of compact namespace because IT is a compact schema is a compact schema the basic namespace and finally the one the basic namespace and finally the one used in the data file used in the data file when the location of the schema used for when the location of the schema used for validation is declared validation is declared since one SDMX schema is also an xml since one SDMX schema is also an xml file to validate IT out of the root file to validate IT out of the root element element different imports sentences are declared different imports sentences are declared to get a reference to all the schemas to get a reference to all the schemas needed needed this also means there are three schemas this also means there are three schemas this xml file is going this xml file is going to be validated against to be validated against furthermore in this schema there are furthermore in this schema there are other specifications other specifications the target namespace is the namespace of the target namespace is the namespace of the data file the data file and is needed at a time of validation and is needed at a time of validation this means that this target namespace this means that this target namespace must be the same than the one declared must be the same than the one declared for the data file for the data file in this example the following in this example the following declaration according to the target declaration according to the target namespace namespace should be declared in the data file And finally the attributes element form default and attribute form default default and attribute form default that should always have the same values that should always have the same values as we see here these two values as we see here these two values are used to indicate whether or not are used to indicate whether or not locally declared elements and attributes locally declared elements and attributes must be unqualified Xslt is an xml based language used for the transformation of xml used for the transformation of xml documents documents into other xml or human into other xml or human readable documents for example HTML readable documents for example HTML the idea is that the original document the idea is that the original document is not is not changed but a new one is created changed but a new one is created using one xslt document using one xslt document based on the content of the original based on the content of the original document document in this example we can see a conditional in this example we can see a conditional sentence sentence which depends on the element code list which depends on the element code list the main element is the conditional the main element is the conditional element element this means that if the expression to be this means that if the expression to be evaluated evaluated test does not have an attribute test does not have an attribute code list the output value will be code list the output value will be extracted extracted from the attribute concept ref from the attribute concept ref and with the literal text type and with the literal text type in this exercise we are going to explain in this exercise we are going to explain how to transform how to transform an sdmx-ml structure file to an HTML an sdmx-ml structure file to an HTML presentation presentation using an xslt file using an xslt file the first step is to open one structure the first step is to open one structure xml file xml file go to the file menu and then click open go to the file menu and then click open and select the structure xml we are and select the structure xml we are going to work with We have our structure file loaded and we have to select have to select the transform action for this file the transform action for this file using the xsl file to make this action using the xsl file to make this action so we go to the xml menu option so we go to the xml menu option and click in xsl transform and click in xsl transform then we select the xsl file then we select the xsl file we are going to work with a new file is we are going to work with a new file is generated generated by the xml copy editor tool by the xml copy editor tool with HTML code We save this new document as an HTML page page once IT is saved we can open this new once IT is saved we can open this new HTML HTML file in any HTML browser to see the file in any HTML browser to see the results results we can access the information of the we can access the information of the sdmx-ml structure file sdmx-ml structure file in a user-friendly format in a user-friendly format in this exercise we are going to see how in this exercise we are going to see how to create an xml schema to create an xml schema from an SDMX ml structure file from an SDMX ml structure file and having the xml data file how we can and having the xml data file how we can validate this data file against the validate this data file against the schema we schema we created according to the previous sdm created according to the previous sdm xml structure file xml structure file the first step consists in opening the first step consists in opening a structure xml file we go to the file a structure xml file we go to the file menu menu and press open here we select the and press open here we select the structure structure xml we are going to work with xml we are going to work with once we have our structure file loaded once we have our structure file loaded we select the action we select the action transform for this file selecting transform for this file selecting the transformation file we need for this the transformation file we need for this action action so we go to the xml menu and press so we go to the xml menu and press in xsl transform here we select the in xsl transform here we select the transformation file needed transformation file needed a new file is generated by the a new file is generated by the application application with an xml code we save with an xml code we save this new document as a schema file this new document as a schema file we can use the schema created to we can use the schema created to validate xml data files validate xml data files for this purpose we open our xml data for this purpose we open our xml data file file we go to the file menu and press open we go to the file menu and press open we select the xml data file we are going we select the xml data file we are going to work with to work with as we can see in the xml data file there as we can see in the xml data file there are some namespaces are some namespaces we have to take into consideration we have to take into consideration xmlns stat underscore sts xmlns stat underscore sts is the namespace that has to be declared is the namespace that has to be declared in the schema created before specifying in the schema created before specifying in terms of validation the namespace in terms of validation the namespace used in the data file to do this used in the data file to do this in the schema we have to add the in the schema we have to add the namespace namespace as follows in the xml data file as follows in the xml data file according to the xsi schema location according to the xsi schema location attribute there are two schemas the xml attribute there are two schemas the xml data file is validated against data file is validated against schema underscore 2 underscore schema underscore 2 underscore validate dot xsd validate dot xsd should be changed to the schema we should be changed to the schema we created before created before the one we created to tell the parser the one we created to tell the parser we want to validate the xml data file we want to validate the xml data file against our schema against our schema if we check the namespace that is used if we check the namespace that is used in the xml data file in the xml data file for this schema we can see that there for this schema we can see that there is a urn path that must is a urn path that must also be declared in the schema created also be declared in the schema created already as already as target namespace inside the element target namespace inside the element xs double point schema xs double point schema to maintain the references now to maintain the references now we can validate the data file we can validate the data file in this example we are going to create in this example we are going to create an HTML file an HTML file from an SDMX ml data file from an SDMX ml data file the first thing we have to do is open the first thing we have to do is open compact data xml file compact data xml file we go to the file menu and press open we go to the file menu and press open where we select the compact data xml we where we select the compact data xml we are going to work with are going to work with for the process we have to change for the process we have to change two values of the transformation file two values of the transformation file before using IT before using IT we add the values of the data file and we add the values of the data file and the DSD the DSD inside the file input inside the file input underscore file is the data file underscore file is the data file we are going to process and look lookup we are going to process and look lookup underscore file is the schema underscore file is the schema we require in this case we require in this case estat underscore Eurostat estat underscore Eurostat underscore sts underscore underscore sts underscore compact compact.Xsd compact compact.Xsd at the beginning of the document we at the beginning of the document we should have these two should have these two lines with no data elements lines with no data elements IT is important to have these two files IT is important to have these two files data file data file and esd in the same folder as the and esd in the same folder as the transformation file transformation file since we are using relative paths since we are using relative paths to find the documents otherwise to find the documents otherwise we have to put the absolute path where we have to put the absolute path where the documents are stored Since we had our data file previously loaded loaded in the xml editor we select the action in the xml editor we select the action transform for this file selecting the transform for this file selecting the transformation transformation file we changed before we go to the xml file we changed before we go to the xml menu menu and select xsl transform and select xsl transform where we select the transformation file where we select the transformation file we are going to work with we are going to work with a new file is generated by the xml copy a new file is generated by the xml copy editor tool editor tool with an HTML code a new HTML page with an HTML code a new HTML page has been created we has been created we save this new document as an HTML page save this new document as an HTML page and we can open this new HTML file and we can open this new HTML file in any HTML browser to see the results in any HTML browser to see the results we have a user-friendly result of an we have a user-friendly result of an sdmx-ml data file sdmx-ml data file web services represent the coming web services represent the coming generation of Internet technologies generation of Internet technologies they enable computer applications to they enable computer applications to exchange data exchange data directly over the Internet essentially directly over the Internet essentially allowing modular or distributed allowing modular or distributed computing computing in a more flexible fashion than ever in a more flexible fashion than ever before before many web services standards already many web services standards already exist exist and there is no need to reinvent them and there is no need to reinvent them for specific use within the statistical for specific use within the statistical community community specifically simple object access specifically simple object access protocol protocol or soap and wsdl or soap and wsdl web services description language can be web services description language can be used by SDMX used by SDMX to complement the data and metadata to complement the data and metadata exchange formats they are standardizing exchange formats they are standardizing since IT becomes important to be able to since IT becomes important to be able to describe the communications describe the communications in a structured way wsdl in a structured way wsdl addresses this need by defining an xml addresses this need by defining an xml grammar grammar for describing network services as for describing network services as collections of collections of communication endpoints capable of communication endpoints capable of exchanging messages exchanging messages IT is important to observe that wsdl IT is important to observe that wsdl does not introduce a new type definition does not introduce a new type definition language language wsdl recognizes the need for rich wsdl recognizes the need for rich type systems for describing message type systems for describing message formats formats and supports the xml schema and supports the xml schema specification specification as its canonical type system here as its canonical type system here we can observe an example of census hub we can observe an example of census hub wsdl file wsdl file all SDMX web services should be all SDMX web services should be described described using wsdl instances using wsdl instances according to the use of wsdl according to the use of wsdl to specify the aspects of this multiple to specify the aspects of this multiple message message exchange which they support exchange which they support to ensure interoperability between SDMX to ensure interoperability between SDMX web services web services compliance with sections of the compliance with sections of the wsi profile 1.1 wsi profile 1.1 is recommended for all SDMX web services is recommended for all SDMX web services in this section some examples of soap in this section some examples of soap envelopes used in SDMX registry envelopes used in SDMX registry are going to be explained two parts are going to be explained two parts can be identified clearly in any soap can be identified clearly in any soap envelope envelope which is the root element of the which is the root element of the document the soap header document the soap header and the soap body and the soap body the soap header includes a custom the soap header includes a custom element named element named security header used for conveying security header used for conveying authentication information authentication information in the form of three strings for example in the form of three strings for example a username a password and a domain a username a password and a domain due to the fact that this information is due to the fact that this information is too sensitive to be sent too sensitive to be sent IT is strongly recommended to enable IT is strongly recommended to enable https https in the server where the web services in the server where the web services the most important part of this soap the most important part of this soap envelope is presumably the soap envelope is presumably the soap body which due to the fact that a web body which due to the fact that a web service service is a document style implementation must is a document style implementation must include a registry interface SDMX ml include a registry interface SDMX ml message message in the example the query is not shown in the example the query is not shown although the web service would respond although the web service would respond with a registry interface SDMX with a registry interface SDMX ml message on the occasion of an ml message on the occasion of an incoming request incoming request there is a chance that something goes there is a chance that something goes wrong before the actual registry wrong before the actual registry interface message response is prepared interface message response is prepared for example connection to the database for example connection to the database is not feasible is not feasible in such cases the web service responds in such cases the web service responds with a soap fault message with a soap fault message one soap request according a wsdl one soap request according a wsdl structure may look like this And the soap response example according to the same wsdl according to the same wsdl structure should look as follows

2023-07-31 04:18

Other news