1. No category

.
. Functional learning .
Reproducing kernel spaces.
how to build them
.
Xavier Mary and Stéphane Canu
[email protected]
asi.insa-rouen.fr/~scanu
.
INSA Rouen -Département ASI
Laboratoire PSI
. . Functional learning .Reproducing kernel spaces.how to build them. – p.1/12
'
&
#$
"
tanh
is NOT a positive kernel : but it works
-
:
)
)
*,
)
+*
or
norms are NOT hilbertian
()
%
"
tanh
$#
!
%
!
Motivations
: is NOT a norm
What do we need to learn ?
. . Functional learning .Reproducing kernel spaces.how to build them. – p.2/12
roadmap
Discrimination par SVM
1
0.8
0.6
0.2
0.4
0
−0.2
how to choose ?
learning in functional space
Reproducing Kernel Hilbert Space
−0.4
−0.6
−0.8
−1
0
0.5
1
1.5
2
2.5
3
3.5
4
Multiscale Approx on Frame
2
how to build kernels ?
injective operators
exemple
1.5
1
y
0.5
how the solution looks like ?
representer theorem
0
−0.5
Original
Data
SVM approximation
Multi resolution
semi parametric
−1
−1.5
0
1
2
3
4
5
x
6
7
8
9
10
. . Functional learning .Reproducing kernel spaces.how to build them. – p.3/12
roadmap
Discrimination par SVM
1
0.8
0.6
0.2
0.4
0
−0.2
how to choose ?
learning in functional space
Reproducing Kernel Space
−0.4
−0.6
−0.8
−1
0
0.5
1
1.5
2
2.5
3
3.5
4
Multiscale Approx on Frame
2
how to build kernels ?
injective operators
exemple
1.5
1
y
0.5
how the solution looks like ?
representer theorem
0
−0.5
Original
Data
SVM approximation
Multi resolution
semi parametric
−1
−1.5
0
1
2
3
4
5
x
6
7
8
9
10
. . Functional learning .Reproducing kernel spaces.how to build them. – p.3/12
interpretation : local measure of the hypothesis
in
close to
/
_ _ _/ Measure at
close to
the measure describes the function
....................
continuous
we can measure the function
............................................
this is not the case of
/
:
_ _ _/ Measure at
coucou
Continuity is NOT a mater of inner product
. . Functional learning .Reproducing kernel spaces.how to build them. – p.4/12
exists
&
is continuous :
and
exists :
Evaluation space
Hilbert space - inner product ..................................Duality
measures ..... functions
helps me to measure the
AND measures
we need two spaces : hypothesis
How the information I have got at point
quality of the hypothesis at point
. . Functional learning .Reproducing kernel spaces.how to build them. – p.5/12
its topological dual
-
)
)
,
)
)
its topological dual
,
-
The duality map :
-
-
-
-
Duality map : an example
-
-
-
DD
DD
DD
DD
!
Dual is seen as a functional space
-
HH
HH
HH
HH
H#
. . Functional learning .Reproducing kernel spaces.how to build them. – p.6/12
*
+ «good » properties
Kernel existence
general case
/
DD
DD
DD
DD
D"
II
II
II
II
I$
Hilbertian case
measures ..... functions
Generalization of reproducing kernels
. . Functional learning .Reproducing kernel spaces.how to build them. – p.7/12
*
*
*
/
FF
FF
FF
FF
F"
HH
HH
HH
HH
H$
are in duality but NOT in
-
and
using injective operators
,
,
(
-
(
-
let’s map them to
-
how to build duality kernels
. . Functional learning .Reproducing kernel spaces.how to build them. – p.8/12
are in duality but NOT in
-
and
using injective operators
,
,
,
and
(
-
(
-
let’s map them to
-
how to build duality kernels
. . Functional learning .Reproducing kernel spaces.how to build them. – p.8/12
are in duality but NOT in
-
and
using injective operators
,
,
,
define the duality map :
and
(
-
(
-
let’s map them to
-
how to build duality kernels
. . Functional learning .Reproducing kernel spaces.how to build them. – p.8/12
how to build duality kernels : an illustration
,
+*
+*
,
,
+*
*
(
-
(
-
injective operators on
. . Functional learning .Reproducing kernel spaces.how to build them. – p.9/12
how to build duality kernels : an illustration
,
+*
+*
with respect to
,
,
,
)
)
+*
*
+*
)
)
with respect to
(
-
*+
*+
(
-
injective operators on
. . Functional learning .Reproducing kernel spaces.how to build them. – p.9/12
how to build duality kernels : an illustration
,
+*
+*
*+
,
,
,
tanh
"
"
to get
and
"
further work : find
!
)
)
+*
*
+*
)
)
with respect to
(
-
with respect to
*+
(
-
injective operators on
. . Functional learning .Reproducing kernel spaces.how to build them. – p.9/12
Back to the Hilbertian case
..(measures are hypothesis ans R.K.H.S.)
..(only one function)
-
*+
*+
(only one operator)
,
*+
*+
+*
+*
,
+*
*
*+
(
Carleman operator
. . Functional learning .Reproducing kernel spaces.how to build them. – p.10/12
Back to the Hilbertian case
..(measures are hypothesis ans R.K.H.S.)
..(only one function)
-
*+
*+
(only one operator)
,
*+
*+
+*
............................................................
to define you have to know
with you define
+*
,
+*
*
*+
(
Carleman operator
. . Functional learning .Reproducing kernel spaces.how to build them. – p.10/12
Back to the Hilbertian case
..(measures are hypothesis ans R.K.H.S.)
..(only one function)
-
*+
*+
(only one operator)
,
*+
*+
+*
: there exits a countable basis such that
is separable so is
-
............................................................
to define you have to know
with you define
+*
,
+*
*
*+
(
Carleman operator
. . Functional learning .Reproducing kernel spaces.how to build them. – p.10/12
The representer theorem
+*
Assume
"
"
*+
is a subduality of
with kernel
convex and differentiable ...
denotes its Gateau derivative)
such that
. . Functional learning .Reproducing kernel spaces.how to build them. – p.11/12
The representer theorem
+*
Assume
"
"
and
(thanks to the convexity of
)
Then for
*+
is a subduality of
with kernel
convex and differentiable ...
denotes its Gateau derivative)
such that
. . Functional learning .Reproducing kernel spaces.how to build them. – p.11/12
The representer theorem
+*
Assume
"
"
and
)
(thanks to the convexity of
"
"
"
Then for
*+
is a subduality of
with kernel
convex and differentiable ...
denotes its Gateau derivative)
such that
. . Functional learning .Reproducing kernel spaces.how to build them. – p.11/12
The representer theorem
+*
Assume
"
"
and
)
null space of
"
"
"
"
(thanks to the convexity of
Then for
*+
is a subduality of
with kernel
convex and differentiable ...
denotes its Gateau derivative)
such that
. . Functional learning .Reproducing kernel spaces.how to build them. – p.11/12
Conclusion
because we want
you can learn without kernel but there is one !
and the continuity of the evaluation functional
a framework generalizing R.K.H.S to non hilbertian spaces
build kernels thanks to simple operators
and the regularizer
"
"
"
"
we know the shape of the solution
gives you the kernel
How to learn the coefficients ? How to determine the cost function ?
. . Functional learning .Reproducing kernel spaces.how to build them. – p.12/12