1. No category

Résumé
On réalise fréquemment des efforts remarquables et
coûteux afin de développer et implanter de nouveaux
systèmes d’information dans des environnements
complexes tels que ceux qui assurent la distribution
des services publics (e-Government). Pourtant, on se
pose rarement la question (voire jamais ?) a posteriori
de l’efficacité de ces systèmes en menant une étude
comparative adéquate avec des situations réelles analogues. Sans fournir cet effort méthodologique
d’évaluation ex post, il y a un risque de continuer à
réaliser des investissements conséquents inutiles tandis
que l’on pourrait augmenter le rendement de ces investissements en cherchant du côté d’autres facteurs qui
amélioreraient réellement ces services.
Cet papier souligne les étapes de la comparaison menée entre des services de support de recherche
d’emploi fournis par deux régions européennes : Lombardie (Italie) et Catalogne (Espagne) et expose les
difficultés objectives de cette analyse. Il compare certains aspects et problèmes rencontrés par les deux différents systèmes, et il met en évidence des cas d’étude
qui enquêtent sur l’impact des deux systèmes sur deux
zones européennes similaires d’un point de vue socio
économique.
Mots clefs :
Plateforme de services pour l’emploi, Systèmes
d’information pour l’emploi, Analyse des charges,
Solutions organisationnelles d’intégration des systèmes, Systèmes de soutien a la décision stratégique.
Abstract
Expensive, large-scale efforts are often made to develop and implement new information systems in
complex areas such as support to public services (egovernment). The effectiveness of such systems is
seldom - if ever - put to the test a posteriori by making
an appropriate comparison with similar set-ups. Without such an ex post methodological evaluation effort,
substantial investments may continue to be made to no
avail, when the services could instead be improved by
simply focusing on other factors.
This paper outlines steps for comparing the job market
place support services provided by two European regions, Lombardy (Italy) and Catalonia (Spain), and
shows the objective difficulties of doing this. It compares some aspects and issues of the two different systems used. Finally, some research questions investigating the impact of the two systems on two similar
socio-economical European areas are outlined.
Key-words:
Job Marketplace Services, Employment Information
Systems, Requirements Analysis, Organisational Issues on Systems Integration, Strategic Decision Support Systems.
The Neglected
Problem: How to
Really Benchmark the
Effectiveness of ICT
Based Services? A
Case Study
Le problème négligé:
Comment réellement
évaluer l’efficacité des
services informatisés ?
Une étude de cas
Mirko Cesarini, Mario Mezzanzanica.
Department of Statistics, University of Milan Bicocca,
Milan, Italy
[email protected], [email protected]
Mariagrazia Fugini, Piercarlo Maggiolini
Politecnico di Milano, Department of: Electronics
and Information, Management and Engineering,
Milan, Italy
[email protected], [email protected]
Ramon Salvador Vallès
Department of Business Organization, Universitat
Politècnica de Catalunya, Barcelona, Spain
[email protected]
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
Introduction
Expensive, large-scale efforts are often made to develop
and implement new information systems in complex areas such as support for public services (e-government).
The effectiveness of such systems is seldom - if ever put to the test a posteriori by making an appropriate
comparison with similar set-ups. Without such an ex post
methodological evaluation effort, substantial investments
may continue to be made to no avail, when the services
could instead be improved by simply focusing on other
factors.
This paper analyses two service provisioning systems
supporting the job market place. The first service system
is located in Lombardy (in Northern Italy) while the second is located in Catalonia (Spain). Both systems largely
exploit Information and Communication Technologies
(ICT) to achieve their goals, but the service provisioning
models are quite different. The Lombard system (named
Borsa Lavoro Lombardia – BLL) connects public and
private operators providing services to job seekers and
employers. In such a system a curriculum vitae or a job
offer posted to a public employment office or to an
agency is shared among all the operators being part of the
federation. BLL leverages web technologies through
which final users can post job offers and demands with
no need for physical a presence in front of operators or
mediators (agencies). Analogously, employers can post a
job offer on the web.
On the other site, the Catalan system is based on a network of public offices not collaborating yet with private
agencies working in the recruitment area. Job seekers are
hence required to visit a public office to submit their curricula and to have them validated. ICT are used in such a
context to notify a person when his/her curriculum raises
the interests of a potential employer. The system is also
useful to communicate to the central Spanish Government the lists of unemployed to trigger unemployment
contributions.
Coming to the features of Lombardy and Catalonia, we
show that the two regions have various similarities from
the demographic, economic, and cultural point of view.
In such similar contexts, the comparison of the two quite
different service provisioning systems stimulates some
interesting considerations about the impact of the ICT
based employment services on the local employment
market and in general about the context. This paper is
structured as follows: Sec. 1 gives the motivations of our
work by presenting some elements that drive the new
requirements for services to employment, presenting the
guidelines for our analysis, and outlining some social and
demographic similarities among Lombardy and Catalonia. These elements define the interest of our research.
Sec. 2 introduces the different public employment services provision models implemented in the two regions;
Sec. 3 discusses some possible reasons explaining the not
exciting performances of the two system services and
reports some comparison indexes. Finally, Sec. 4 draws
some conclusions and illustrates future work.
1. Analysis guidelines
The purpose of this section is to draw our analysis guidelines useful when comparing the labour markets of Catalonia and Lombardy, and the tools and services existing
in two fairly similar geographical and economical areas
such as Catalonia and Lombardy.
The idea is to investigate, through employment market
emerging requirements (at least the most prominent ones,
currently) and through a comparison of economic, social
and demographic indicators if different sets of ICT-based
services are significantly similar or different in performance when used in similar contexts. If they are different,
we want to explore why it is so, and validate if for example the influence of ICT is irrelevant, or if the reason is
that innovative ICT-based services (e.g., based on Web
Services) require conditions of use that are not being met.
The analysis should evidence what these conditions are,
namely if cultural factors are more important than technological ones.
Our analysis points can be shown in a schema as depicted
in Fig. 1.
Figure 1 Analysis Schema
1.1. Requirements and Challenges of the
Job Marketplace
Market unpredictable trends and high demand variability
have undermined firms stability in recent years. Firms in
turn have reorganized their activities to improve competitive capabilities by focusing on innovation and flexibility. Such phenomena have had a strong impact on the job
market place and on the occupational dynamics, originating high job mobility dynamics, unforeseeable in the past
years. For example, workers change job more frequently
both in Catalonia and Lombardy, with respect to some
years ago. In the Milan province (the main province from
the economical point of view in Lombardy), the population changing job or beginning to work as a share of the
active working population are 16-18% in 2001 and 3237% in 2006 (the significant information is the indicator
trend rather than the exact values) (Mezzanzanica,
Lovaglio, 2008). Similar indicator values hold for the
whole Lombard provinces. Several groups can be identified among the changing-job-population: people that can
collect the opportunities provided by the market and
therefore can easily improve or stabilize their working
conditions (80% of the observed set); people stable in
bad contractual conditions (e.g. flexible contracts) or
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
worsening their career path. The latter group requires
help or support during the job change phases. For the
latter group therefore it is essential the design and development of instruments, institutions, and active policies
helping such people addressing issues that prevent an
improvement of their working conditions. Information is
essential for such support, as well as the empowerment of
network of service providers (built through the cooperation of several job market place actors) that can rapidly
react to the market evolution and to the user needs.
A second important reason motivating the need for employment services and their analysis in different realities
is that public and private institutions can exploit several
channels to deliver services for the job market place.
Web based portals are used to collect job demands and
offers, curricula and related documents. Teletext
(http://en.wikipedia.org/wiki/Teletext), sms, phone gateways can be used as well, although these media are
mostly used as support for information sharing activities.
The aim of these media is to share information and to
push job seekers and employers to access the web sites or
the public and private institution premises.
1.2. Economic, Social and Demographic
Comparison of Lombardy and Catalonia
The Lombardy and Catalonia Regions are very similar
from the economic and the cultural point of view. Both
regions are members of the “Four Motors of Europe”
association (4 Motors,1988) whose members are four
highly industrialized regions in Europe (the association is
also composed of Rhône-Alpes in France, and BadenWürttemberg in Germany) significantly contributing to
the European GDP. Lombardy and Catalonia are the
main contributors to their country’s GDPs. Furthermore,
they are considered playing a connective role between
southern and northern European regions and Countries.
Lombardy
Area (Km2)
Population
(thousands)
Density
23861
9434
395/
Km2
Lombardy /
Italy
7.9%
16.14
%
Catalonia
Catalonia /
Spain
32627
6860
6,45%
15.94%
210/
Km2
Table 1: Geographic and demographic data. Source: (Eurostat,
2005).
From the historical and geographical point of view, both
Lombardy and Catalonia are located in the Latin Europe
(the Romance-speaking nations of Europe, since their
language stem from Latin) they have similar religious
distribution and strong cultural similarities. The economic similarities are investigated in the next subsection.
1.3. Demographic and Economic Indicators
Table 1 shows some geographic data describing Lombardy and Catalonia which are useful for our comparisons.
Table 2 shows some macro economic data. Lombardy
and Catalonia are the main contributors to their country
GDP with respect to the other regions.
Lombardy
GDP (106
€)
GDP /
Population
298,285
Lombardy /
Italy
21%
30,566.9
Catalonia
Catalonia
/ Spain
170,226
18,83%
27,345.5
Table 2: Macro-economic data. Source: (Eurostat, 2005), Regional Gross Domestic Product, Eurostat Statistic Year Book
2006.
Lombardy and Catalonia are top ranked regions within
their Countries considering the Export and Import quotas,
as shown in Table 3. The distance between the second
topmost region (Veneto and Comunidad Valenciana respectively) are notable.
Lombardy
Veneto
Catalonia
Comunidad Valenciana
Export
27.9%
14%
26.6%
11.8%
Import
37.2%
11%
29%
8%
Table 3: The two top ranked regions for Export and Import in
Italy and Spain. Data about year 2004, source: (D'Intinosante,
2005), (Comercio exterior, 2005).
The two regions show similarities also with respect to
their economic sector composition, as shown in Table 4.
Lombardy
Catalonia
Agriculture
1.9%
2.5%
Industry
40.1%
38.5%
Services
58.0%
58.9%
Table 4: Economic sectors composition. Source: (European
Commission, 2004).
Job market places indicators such as: 1) employment
rates by sex and age and 2) employment by fulltime/part-time and sex confirm that the Lombard and the
Catalan Job Market place are quite similar (Eurostat,
2006) (Eurostat, 2006a).
2. Public Employment Services
Provisioning Models
In this section, we come to observe the job marketplace
in more depth. Both Lombardy and Catalonia provide
services to the job market place but in different ways.
Such two different models are described in the next subsections.
2.1. The Institutional Mode of Managing
the Catalan Employment Marketplace
The public administration in Catalonia provides services
for the labour market through the SOC (Servei
d’Ocupació de Catalonia), which has a network of 73
public employment offices called OTGs (Oficines de
Treball de la Generalitat). The SOC is an organization
that belongs to the Catalan Ministry of Labour and its
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
aim is to integrate and manage employment policies. It
provides services for job seekers and recruiting companies with the aim of improving the efficiency of the labour market and increasing the level and quality of employment.
and Administration, as well as interoperability between
public departments. It develops existing networks of
communication between departments (such as SARA
network) and that of integrated network of service to the
citizens.
The SOC’s Information System provides support for the
registration, monitoring and management of the payment
of benefits to unemployed people who have signed on at
public employment offices, for the registration and monitoring of contracts, for job seeking and recruitment activities and for activities related to collaborating and
training organizations. The Information System retrieves
and stores job seekers’ CVs, job offers, vacancy descriptions, employment contracts, information on training
courses, economic statistics, information on employment
and links to information sources. Information system
users share such documents and information according to
privacy restrictions. Job offers are posted on the office
notice boards and in turn matched with locally available
CVs by staff members, and later job seekers’ contact
details are passed on to companies.
This law was designed within the Spanish plans to reach
the objectives of the Lisbon Agenda of the European
Union (year 2000). These plans aimed to improve technological innovation and enhance information society
through the Ingenio 2010 Program (November 2005), and
the Avanza Plan included in it, in order to develop Digital Public Services, among other objectives.
In addition, the SOC website provides job seekers with
selected information on job offers, including the position
description, the skills required and the deadline for submitting an application. Nonetheless, job seekers must go
to an OTG in person to carry out any employment-related
procedures. A limited number of online procedures (e.g.,
checking CVs and reports) are offered to those who have
been registered at an OTG. The SOC website provides
additional information such as guidelines on how to write
a CV, news, links and training courses. It also features
information for firms on training courses, employment
contracts and employment promotion actions, and provides services that enable firms to upload, publish and
manage job offers and to furnish the SOC with data on
employment contracts (being the firms already registered
at an OTG). The SOC website also offers information
and services to collaborating bodies and training centres.
Private agencies are also active participants in the job
market. Their job is to provide companies with services,
which include selecting potential employees using psychometric tests and conducting interviews. These agencies have their own databases of CVs for some positions,
although there is some outsourcing of the task of matching CVs with profiles to e-recruitment companies such as
infojobs.net, which allows them to cut the cost of posting
offers and matching CVs. The use of private recruitment
websites is quite common in Catalonia, infojobs.net being the most popular. Other popular websites are infofeina.com, monster.es, trabajar.com, laboris.net and
infoempleo.com.
In Spain, the "Law on Electronic Access of Citizens for
Public Services" was enacted in June 2007. This law
gives the opportunity to citizens to get electronically
linked with the Public Administration and sets a deadline
(December 2009) for a full implementation. It establishes
mechanisms to ensure the relationship between citizens
The Information System of Public Employment Services
(SISPE) is established by a Spanish law dated December
2003, and its main objectives are: a) to provide information for the management of active employment policies,
(under the responsibility of the Regional Administrations), b) to support the provision of benefits for unemployed people (under the State responsibility), and c) to
integrate information for statistical analysis and management. This system replaces the SILE (Information
System of Labour Employment), in May 2005.
In Spain there are two models for the management of
public employment services and thus two models of IS.
While some regions use the IS of the Spanish Public Employment Service, sharing applications, and DBs, other
regions have its own IS connected to that of the State to
share some data. The SISPE supports the integration of
the different systems. Catalonia has its own IS called
SICAS, which is connected in real time to SISPE.
2.2. The Lombard Interconnected Job
Marketplace System
The Italian local Public Administration “Regione
Lombardia” autonomously created a federation of local
Portals for employment services. Each Portal was directly
managed by the local PAs (typically, Provinces), and had
no or few relationships, mainly administrative oriented,
with peer local Portals. In a second phase, local Portals
joined a federation created to establish a collaboration
network among the local Portals (CENSIS, 2006). The
Italian Employment Information System was conceived
according to the requirements drawn by the Italian law
dated October 23rd, 2003, which liberalized the employment market-place from the public monopoly, but required the intermediaries (both public and private) to
share CVs and Vacancies through a federated information
system. The federation was conceived as glue of the local
nodes (Borsa Lavoro Lombardia, 2006; SINTESI, 2005),
more back-end than front-end oriented. As a result, all
the local Information Systems are connected into a distributed and cooperative Information System (Coulouris,
Dollimore and Kindberg, 1994).
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
Figure 2: Network structure of the Italian Employment IS.
Moreover, since most local Portals (and the corresponding local Information Systems) are heterogeneous, both
from the technological and organizational viewpoint,
they have been connected through an interoperability
framework that allows to build a Cooperative Information
Systems. Such a federative approach, although simplifying the connection of sites, e.g., leaving local databases
and applications unchanged and interconnected through
applicative gateways, raises issues concerning the management of global functions, among which data access
policies, service interoperability, data/service mediation,
and security. BLL is a node of the National Federation.
The federation appears as a multilayer structure organized according to a geographical basis (towns and cities,
Provinces, Regions, Country), see Fig. 2. The external
layer is composed of the local domains, grouped and
linked under the direct control of a provincial domain.
The middle layer is composed by regional nodes (each
regional node is the reference node for the provincial
domain present in the region). Each regional Portal is
connected to peer regional Portals; finally, all nodes refer
to the national domain. This multilayer structure exploits
the web service technology, used for exchanging data
between applications and systems running on different
platforms. The federated system is fed by information
provided by the end-users. Specifically, Job Seekers and
Employers. Private and Public Job Agencies (which play
as the mediators of the job matching procedures) are the
front-end of the system to the final users. Since the network relates also to education, training and job requalification programs, Schools and Universities, and
other PA (Pension Registries, Social Security Offices and
the Ministry of Welfare) are also nodes of the distributed
Information System.
We briefly outline the Job Seeker typical interaction with
the system in order to show the service model. 1) The Job
Seeker inserts his/her personal data and the Curriculum in
one node of the network, usually the one he/she wants to
be registered in. 2) After registering and after providing
his/her credentials, the user fills in the fields with all the
mandatory and confidential personal data, which are
electronically protected and visible only upon explicit
consensus by the user who remains the data owner. 3)
The job seeker completes the Curriculum section of the
web site loading his/her curricular data. 4) The curricular
section is split in two portions: a public and a private one.
Public curricular data are shared in the network and used
to compute the match against available job offers and for
statistical purposes. Contact and registry data are not
present in the public curricular data and are the main
content of the private portion. They are shared among
nodes according to business rules and with the consent of
the owner. Public curricular data are mainly intended as
information to be shared among entities of the federation
(such as private job agencies). Contact data are exchanged only according to business rules to preserve the
private agencies business. A registration and authentication phase is required also for the Employer, who loads
confidential data and the job profile requested to apply
for the vacancy. 4) The local system archives both job
offers and requests in a local database. The cooperative
Information System queries local databases to return the
best CV/offer match according to job search and vacancy
criteria. 5) Once an applicant is considered suitable for
the vacancy, the public curricular data are sent to the
Employer by means of notification systems (generally
Publish&Subscribe mechanisms, emails, short message
systems, and so on). 6) The Employer chooses the best
candidates and decides about a possible selection; afterwards, the mediator contacts the Job Seeker to propose
him/her the selection. 7) If the Job Seeker accepts and
explicitly gives his/her ok, personal data are sent to the
Employer (or to the Agency) for a selection interview. 8)
If the selection ends successfully with a hiring step, the
system communicates mandatory information about the
worker’s contract to the Pension Registries, which update
the job status of the hired person.
The end-users may also search for training opportunities
and job re-qualification courses, e.g., in case of long-term
unemployed people. In that case, the end-user can browse
the federated Portals, filling in the required fields with
the search criteria and starting the research. Schools and
Universities, after registration, can insert both training
opportunities and job re-qualification courses and are in
charge of providing all information useful to apply for a
job.
3. Possible Reasons Explaining
Different Roles and Similar Performances
The entities providing services in the Catalan system do
not share data while information sharing occurs in the
Lombard system. The BLL technical-organizational system should provide better performances concerning the
matching among job offers and demands with respect to
the Catalan system, since the former exploits the supposed benefits of “coopetition”.
The term coopetition is used in management literature to
refer to a hybrid behaviour comprising competition and
cooperation. Coopetition takes place when some actors
cooperate in some areas while compete in some others.
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
Some authors (Brandenburger, Nalebuff, 1996) (Gnyawali, Madhavan, 2001) (Lado, Boyd, Hanlon, 1997)
have recently emphasized the increasing importance of
coopetition for today’s inter-firm dynamics, however
scientific investigation on the issue of coopetition has not
gone much farther. In (Dagnino, Padula, 2002) the authors acknowledge the weaknesses of conventional approaches and underline that coopetition is an underresearched theme. We claim that coopetition is an important research topic in information systems integration as
well and represents a promising solution in many cases.
The BLL system should make (or better, should have
already made) employment services much more performing with respect to the previous systems, and with respect
to similar solutions where public and private operators
exploit disjoint channels (hence with no "coopetition"). A
simulation based on game theory showed that political
incentives (technical, organisational, or economical) can
support the true achievement of the promised benefits.
However, the real results (an evaluation has been carried
out in (Cesarini, Fugini, Maggiolini, Mezzanzanica, and
Nanini, 2007) turned out to be less encouraging since
now. In that paper three hypothesis explaining the gap
between the expected and the real results have been
tested. Shortly, the three hypothesis are: 1) the employment intermediation within the Italian economic system
do not operate only according to a market logic but rather
according to a logic of clans; 2) BLL, which should favour the market logic, has been conceived essentially
according to a governmental/bureaucratic logic not satisfying the customers; 3) the technical and economical
mechanisms underlying the coopetition among public and
private operators are not sufficient when not accompanied by a spirit of reciprocity with no immediate returns,
and by a civic spirit towards the public institutions managing (and guaranteeing) the system.
Concerning the clan logic, a questionnaire distributed
among some ICT companies showed that such behaviour
is present within some Italian companies, although the
questionnaire doesn’t allow to guess the extent of this
behaviour nor allow to guess how this behaviour can
affect the BLL customers. The topics will be further investigated through ad hoc questionnaires.
Concerning the centralized/governmental issue evaluation, the authors observed that new job seekers and employers join the BLL continuously (a registration is required before using the BLL services) and the registered
people continue using the BLL services regularly. Considering the amount of monthly logins and web pages
visited by each registered person or company, it can be
argued that the BLL services are extensively exploited by
the users, thus the initial hypothesis that BLL has been
conceived essentially according to a governmental/bureaucratic logic not satisfying the customers has to
be refused.
Concerning the lack of spirit of reciprocity (shortly: the
lack), game theory showed that such a lack could affect
the BLL performances. Little amounts of lack would
have no effects on the BLL, ad hoc policies ruling the
BLL could balance greater lacks having effect on the
BLL, and for big too big lack, no policy can balance it,
thus, the system is going to be under-used.
Both the Lombard and Catalan system show not exciting
performances about the user exploitation. Several reasons
can be put forward for such low performances. The most
sounding rationale (although still to be fully verified) is
that many people prefer exploiting interpersonal relationship networks rather than public/private agencies providing services for matching job seekers and employers.
The exploitation of such relationships can’t be easily
measured or compared; anyway its contribution can’t be
neglected.
3.1. BLL Specific Issues
The BLL system performances can be related to the poor
habit of end users (namely job seekers and employers) to
interact with a (web) system and to a scarce habit of the
employment offices (both public and private) in advertising the potentialities of an interconnected system.
Moreover, people might distrust the information sharing
features provided by BLL since a potential threat to privacy is felt. Moreover, the not exciting performances of
BLL can be a further reason of scarce use. Although procedures for preventing uncontrolled information disclosure are enacted (e.g. a person currently hired but wishing to find a different job doesn’t like to have her/his
research notified to her/his boss).
Further, it can be observed that the BLL user interface
towards the final users can be ameliorated in terms of
usability (customer interfaces), content organization, and
simplification of the used vocabulary (for professions,
skills, and so on).
As another issue, we notice that the BLL system has no
information whether a match among a job demand and a
job offer turns into a successful hiring or not. In fact, the
BLL system is an information sharing system and has no
connection with the public administrations in charge of
monitoring the people career status (e.g. the Retirement
agency). As a consequence, BLL has very few feedbacks
about its usefulness.
The system complexity can be a reason for the BLL low
performance; such complexity is not only related to technology, but mainly to the massive amount of contents and
to the organizational dimension of the involved actors
which can have difficulties in adapting their internal
processes to the BLL requirements. For example, a private job agency may de facto not exploit the BLL features due to the changes required to the agency internal
processes.
3.2. SOC Specific Issues
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
The performance of public employment offices web system is largely determined by the objectives and organization of the institution. This web system has been conceived as a new channel to complement existing network
of offices. It has some restrictions, however, and people
often have to go to the employment offices and stand in
queues in order to complete related processes, despite
having used the website.
The usability, arrangement and the vocabulary of the
website’s interface are fit for the purpose, although they
could be improved. Furthermore, a digital certificate to
access the services ensures privacy and security, so that
the website’s use is encouraged. Nevertheless in some
cases it is not possible to search for specific job profiles
and obtain all the information available about the job
offered. Moreover, it is not possible to upload CVs via
the website and the public offices’ staff has to feed information into the system using an established format.
Since many of the procedures have to be processed in
person, using the website is not seen as a major issue and
does not make a significant contribution to the system’s
overall performance.
The website is little known among citizens, people are
not accustomed to using it, and the users’ perception is
that the public system offers only administrative advantages rather than increasing the number of contracts. Further more, matching services are not very efficient, the
results for job seeking are uncertain and there is no information about the number of contracts actually secured.
In addiction of that, some contracts are for low-skilled
work and for people who need a certificate from the authorities to prove that they are looking for an employee.
This situation makes users more inclined to use more
efficient systems rather than the public one.
As the systems used by participants (public and private
employment agencies, companies) in the labour market
are not fully integrated among them, there is space for a
greater automation of administrative processes, a better
coordination of the tasks and overall improvements in the
performance. In conclusion, all these facts do not contribute to the system’s performance.
Table 5 attempts an analysis of factors that most influence the positive and negative impact on the marketplace
of the two systems. The classification of factors is taken
from the classical approach from the theories of resistance to information systems implementation. The limited success obtained by both systems lays, in our opinion, in a cultural lack both from the citizens’ side and the
companies’ side in the two Regions, that negatively influence the trust posed by users have towards ICT systems, service based or not, no matter. People still prefer
to use personal social networks to find a job. This could
be a reason. However, we have documented a large access to the BLL system by private agencies, which were
however obliged by a regional law to join the BLL system, and later found out it was a good practice. Public
services, such as the Provinces in Lombardy, still use the
BLL since the system is a communication platform, and
also, unfortunately, a primary way to process administrative data related to job changing. The BLL monitoring
system is no way enabled to provide a true feed back on
the percentage of contacts that ended up to be successful.
In Catalonia, the problem seems to be that SOC, besides
being a mere vehicle of data towards the National system
for unemployment financial support management, offers
scarce individual services (CV and offers cannot be uploaded but are mediated by Public Office employee), thus
preventing its usage from the job seekers and companies
in an autonomous way, besides excluding the private
market as a default design decision. And still, BLL and
SOC present apparently the same level of popularity in
their relative regional areas, as testified by statistics reported in (Della Valle, et al. 2007). And both are being
integrated in the European network of services connected
to EURES.
BLL System
SOC System
System  Usability can be  Usability, arrangement
improved (–)
and vocabulary of the
related
factors  Content organization website’s interface (‡)
can be improved (–)
 Massive amount of
contents (–)
 High system complexity (=)
 No information about
matching results (–)
Users
related
factors
 Digital certificate existence (‡)
 Job information is not
fully provided (≡)
 Upload of CVs is not
possible (≡)
 Many procedures have to
be done in person (≡)
 Poor habit to use the  Website little known (≡)
system (=)
 People not accustomed
 No advertisement of
to using it (=)
the system potential
 Perception that:
(=)
 Distrusting information sharing (–)
o Only
administrative
benefits provided (=)
 Perception of low
performance of the
system (–)
o Matching services not
efficient (=)
o Uncertain results of
job seeking (=)
o Contracts for lowskilled work (=)
o Used by who needs an
administrative prove
(=)
Interac  Important changes are  Very low integration of
required for systems
the systems among parttion
integration
among
ners (=)
related
partners (≡)
factors
 Users role changing (=)
 Users role changing
(≡)
Table 5: Factors influencing the web system effectiveness in both models. Legend: (−) low negative influence,
(=) medium negative influence, (≡) high negative influ-
The Neglected Problem: How to Really Benchmark the Effectiveness of ICT Based Services? A Case Study. 14ème congrès de l’AIM
Cesarini M., Fugini M., Maggiolini P., Mezzazanica M., Vallès R.S.
ence, (+) low positive influence, (‡) medium positive
influence.
4. Conclusions and Future Work
Lombardy and Catalonia have developed very different
services systems supporting their local job market places.
Despite such differences, the social and economic similarities between the two regional areas pave the way for a
comparison of the two service systems. Moreover, both
systems show modest performances, thus they do not
attract a large amount of users as it would be expected.
Several reasons can be used to explain such results, e.g.
the priority of interpersonal relationships with respect to
intermediated demand and offer matching processes, or
the scarce trust in the system security measures to prevent privacy disclosure, or the complexity of using and
maintaining such systems in a lively status. As a future
work we plan to analyse such low-performance-issues in
a deeper way exploiting statistical data that will be available from the two systems and with reference to their
integration in the Eures/SEEMP systems at the European
level.
References
Brandenburger, A.M. and Nalebuff, B.J. (1996), Coopetition, Doubleday & Company, New York
Censis (2006), 9th Report on digital Cities (in Italian)
http://www.censis.it/
Cesarini, M., Fugini, M., Maggiolini, P., Mezzanzanica,
M., and Nanini, K. (2007), “A Solution to Knowledge Management in Information-Based Services
Based on Coopetition. A Case Study Concerning
Work Market Services”, in Interdisciplinary Aspects
of Information Systems Studies, Physica-Verlag HD,
2008, pp. 189-195
Coulouris G., Dollimore J., and Kindberg T. (1994), Distributed Systems, Concepts and Design, AddisonWesley.
Comercio exterior de mercancías. Desglose por cc.aa. y
provincias (2005), Secretaría de Estado de Turismo y
Comercio, con datos de Aduanas.
Dagnino, G. B., and Padula, G. (2002), Coopetition
Strategy a new kind of interfirm Dynamics for Value
Creation, EURAM – The European Academy of
Management, Second Annual Conference - “Innovative Research in Management”, Stockholm, 9-11
May
Della Valle, E.., Cerizza, D., Celino I., Estublier, J., et
al., SEEMP: A Semantic Interoperability Infrastructure for e-Government Services in the Employment
Sector, in The Semantic Web: Research and Applications, LNCS, Vol. 4519/2007, pp. 220-234
D'Intinosante, A., Giorgio, F. (2005), Statistiche relative
all’import/export italiano, Elaborazioni osservatorio
economico MAP su dati ISTAT.
European Commission (2004), Third report on economic
and social cohesion.
http://ec.europa.eu/regional_policy/sources/docoffic/
official/reports/cohesion3/cohesion3_en.htm
Eurostat (2005), Regional statistics. Regional demographic statistics indicators and Population main
demographic indicators, Statistical Office of the
European Communities, http://ec.europa.eu/eurostat/
Eurostat (2006), General and regional statistics. Regional labour market statistics. Employment by sex
and age, at NUTS levels 1 and 2, Statistical Office of
the
European
Communities,
http://ec.europa.eu/eurostat/
Eurostat (2006a), General and regional statistics. Regional labour market statistics. Employment by fulltime/ppart-time and sex, at NUTS levels 1 and 2, Statistical Office of the European Communities,
http://ec.europa.eu/eurostat/
Eurostat (2006b), General and regional statistics. Regional labour market statistics. Unemployment rates
by sex and age, at NUTS levels 1, 2, and 3 (%), Statistical Office of the European Communities,
http://ec.europa.eu/eurostat/
Eurostat (2006c), General and regional statistics. Regional labour market statistics. Long term unemployment (more than 12 months), at NUTS levels 1
and 2, Statistical Office of the European Communities, http://ec.europa.eu/eurostat/
Eurostat (2006d), General and regional statistics. Regional labour market statistics. Economically active
population by sex, age and highest level of education
attained, at NUTS levels 1 and 2, Statistical Office of
the
European
Communities,
http://ec.europa.eu/eurostat/
Gnyawali, D.R. and Madhavan, R. (2001), “Cooperative
Networks and competitive Dynamics: a structural
embeddedness Perspective”, Academy of Management Review, 26(3), pp. 431-445.
Lado, A.A., Boyd, N. and Hanlon, S.C. (1997), “Competition, Cooperation, and the search for economic
Rents: A syncretic model”, Academy of Management
Review, 22(1), pp. 110-141.
Mezzanzanica, M., Lovaglio, P. (2008), “Numeri al
lavoro. Il sistema statistico del mercato del lavoro:
metodologie e modelli di analisi” Quaderni
dell'Osservatorio Mercato Del Lavoro, nº 3, Franco
Angeli.
Sintesi
(2005),
http://www.sintesi.provincia.milano.it