Technology and geography in a paradigm shift: the case of Critical & Conflict Materials in ICT
This first paper we have other stuff on on the general topic is with Andreas Diemer ex-phd student now a very uh promising researcher in the University of at the University of Stockholm. Richard Perkins that is my department and Axel Gross that is an undergraduate student in my department so it's a very explorative study I will let's try to I mean I will try to to go you know on the general picture more than on the detail is very descriptive and uh you know there is no real need to spend time on details so I would like to have feedback on the general idea so I will give a bit of context very short background of the literature because the problem is that it's highly multi-disciplinary so uh we are still digging into various streams the idea why we are interested in that very shortly data methodology and then show you some very preliminary uh descriptive on the technological side and the geography of technology side. So the context is the fact that the mining of critical raw materials that includes also the famous conflict minerals and I will later undefined precisely and that combination their use in many you know electrical electronic advanced electronic products that we know provides a very important material infrastructure to this shift between industry 3.0 industry 4.0. However very little is known on the technological and geographical networks between the advances of technology in this case in particular we had to focus on one and so we selected ICT and that part of artificial intelligence technologies that are part of ICT are a subset of ICT and the demand the technological demand of critical and conflict materials. This has a number of important implications for socioeconomic but also
you know geostrategical consequences in regions in countries government firms and you know these resource dependent value chains. One example just to give you the idea of what I'm talking about you know the Cobalt demand for lithium-ion battery is expected to double in three four years and you see the import exports I mean are involving some of the most troubled countries in the world. This is just a snapshot obviously we could expand far more in this direction. The background literature okay that has looked at critical and conflict materials is really interdisciplinary and by interdisciplinary I mean not only social sciences okay but also a natural sciences, so for this work we had interacted a lot with a group of geologists in the british geological survey because we really needed to understand a bit more uh being you know a poor economic geographer really doesn't grasp the complexity of in fact we were also using the wrong terminology at some point so this help helped us a lot. So one stream of literature look at social economic environmental political consequences sourcing these related minerals or materials. Another stream of the literature is characterized as material criticality looks at the security of the supply precisely at supply chain disruption in various senses. Very related is also the literature on material flow analysis that basically maps stock and flows of minerals their production-consumption across time and space, and finally there is also the huge literature on global production network global value chains attached flagships in terms of the controller multinational corporations and their geography and this obviously has looked at this literature into the extractive sectors and the backward forward linkages that global value chains in this resource-intensive industry can or cannot generate okay. So this is the background we are still you know consolidating that. Why this work?
our concern is really innovation so what is an old idea that I had in mind actually because somebody an old you know friend at SPRUE that is not there any longer we were talking about it a long time ago we were thinking oh you know invention of two new technologies that are somehow implicated that rely on this uh critical materials so we focus on innovation we we look at patterns okay is a patent based analysis but this allows us to explore the relationship between technological change and technological demand for some of these elements natural elements. And also shed light at the same time on the geography of this demand and how this overlaps or not with the with the geography of the supply of these resources. This is an exploratory study that tries to give a preliminary answer to the following question to what extent the ICT-based paradigm relies on technologies that are related to these materials, how this intensity of critical and conflict materials change over time in ICT and what is the organizational national subnational geography of in this you know inventive activities or better the ownership of of this inventive activities within the field of ICT and how this compares with the geography of the supply. So the methodology these are the main steps very very quickly. Okay we use a tax mining okay natural language processing on USPTO data
using the full descriptive text of all patterns granted between 1976 and 2017. Then try to limit our selection of both what we mean by ICT AI and, on the other hand, conflict and critical materials and we carry out a very simple analysis of absolute relative frequencies of keywords appearance for each of the very detailed technological classes. Obviously, these frequencies, are as I said, absolute relative but we believe that the evidence we have some validation also of this methodology rough at the moment. But somehow comforting evidence shows that these patterns might be an effective way to directly capture this relationship obviously this is a very descriptive paper we don't claim causality we are just talking about association and then we also extend the analysis of path to patent citations because patents in ICT AI obviously realize on other patents that are from other technologies that might be intensive or related to conflict and critical materials that and that is an indirect relationship across technologies and then we carry out a very simple micro level analysis of patent acini and looking at this geolocalization to explore the geography of the technological demand compared to that of supply so critical and conflict material definition in this paper we selected the four officially officially defined as conflict minerals that is the famous 3TG - Tin, Tungsten, Tantalum and Gold this is what the the regulation of the world level defined as uh conflict minerals cobalt is not officially considered conflict minerals but we selected it for a very similar characteristic to the 3TG and lithium same story for some characteristics of also concentration of the supply across the world and relationship with human rights abuses exploitation and of of human rights in general. So these are the six selected then what we did is also to validate this yeah this oops
i don't know why it's jumping we try to validate this uh approach of keywords because because obviously we have a problem of false positives right because there might be that some patterns actually the inventor you know report the fact that he or she is looking at a way to substitute this materials and therefore we investigate to what extent let's say global trends in keywords appearance predict global production okay of each of these materials between the 70s and 1917 2017 so you can see then this is we run a regression and you know this these charts gives you the the the linear fit between let's say the the the logs of production uh and the keyword uh occurrence for the for the for this period and you can see that i mean the the match is quite good it's comforting our idea that this method is capturing something that is very very aligned i mean you can see that our measure seems to fit the production data quite well okay with the with the um with the exception maybe of tungsten that is not so aligned but generally speaking data on production of this each of this material comfort that we are not picking up totally a random phenomenon for the ICT an ICT related artificial intelligence we follow the methodology that is there first starting with the wipo definition then using oecd classification and then identifying AI patents in ICT following two quite well-known approaches in order to focus our analysis to all ICT and distinguishing within ICT that part of ICT that is also is ai so uh generally speaking these are just you know a charts that show you that particularly since the end of the 90s there is a surge in patenting activity particularly related to the classes that covers obviously uh communication information technologies and this shift towards uh um towards uh more and more i mean electronic intensive kind of invention and innovation activities now you can see on the other charts that by selecting the first the top uh the top classes excluding the big section of c which is metallurgy and obviously is intrinsically more related to the minerals okay and so we can see that the the most the highest related to these minerals and materials are in fact all classes that are within ICT so um this is just to show you again i have a hundred of these these are the relative frequency of selected keywords by IPC4 class so detailed technological class and you can see by technology these are supposed to be the top 40s obviously we have also frequency by uh each of these materials or absolute because some can obviously uh include more than one keyword so more than one material mineral material so anyway you can see that there are and this i mean at the top are all ICT technologies so what we ask very simple in a very simple way to look at if ICT is different from all other technology in the use in reliance on this specific keyword we run very very simple uh uh you know we examine basically if there is a statistical significant difference between ICT the blue line and all the other technology green and then we also look at AI technologies that are within ICT separately okay so you can see that uh basically you know dividing up the calculating conditional means of relative frequently dividing uh broken down this conditional means in an interval of five years also to look at the how this relation changed over time and the ninety percent interval confidence a confidence interval can can tell you you know in order to to compare across groups and within the groups and what we we get is in fact that ICT seems to be a substantial significantly different from all uh all the other technologies and that is not dependent on the subset of ai within ict okay use keywords more intensely and they use keywords more intensely over time than any other technology and particularly this relationship is stronger since the uh 2000 uh this is okay this is another way this is the citation analysis uh this is a a a chart that refers only we in the subsequent analysis of citation we focus on the most recent period so 2000 2017. what here is reported is just that this is backward citation from ICT to other technologies and what the graph seems to suggest is basically that the the the subclasses with higher centralities that are those in blue actually use uh critical and conflict material more intensely and therefore it would seems that ICT technologies when citing non-ICT patents prefer technologies that are intensive according to our definition in critical and conflict materials so for the citation analysis what we do is i mean we follow that the classical methodology of uh whole Jaffe and Trajtenberg for taking into account the fact that over time all patents stand obviously to cite more prior art so more and so we basically uh demean citation rates we wait for for uh for the average number of citation by class so this is really very very uh well-known stuff but what what is is interesting to see is that uh uh the the the this is the marginal effect of at least one keyword free frequency on the citation rate and what we see is that the higher the relative frequency of critical and conflict mineral keyword appearance in technology the more likely is this technology to be uh cited by the aggregate ICT technologies now this relationship grows quite strongly uh up to the 2005 but then tend to diminish to uh uh 2015 but remaining positive okay well for the other technologies obviously this relationship is is not positive although you can see that tends to be insignificant at least but um in more recent times so um geographical uh geographical uh is uh uh what what's the time i'm losing a bit track of the okay um the geographical analysis the for the geographical analysis obviously we want to look uh for the moment we have looked at the patentastini so we want to just because we were interested in knowing who owns the technology more than who invents the technologies that is a subsequent step that we might want to explore later on and so uh there is obviously no surprise in in seeing that these are the top 25 but in the whole list we we see the big tech okay so the many familiar electronics and so big tech primarily located in japan japan is the country more more present followed by the us and south korea so the usual suspect okay Samsung IBM Canon Micron Sony uh even Apple is there actually and uh few chemical a pharmaceutical giant you can see here i don't because if I don't have a Dupont for example Bristol and Miles and there is also Pfizer there these are the big pharmacan pharma that as we know are the most uh as also i remember for from my corridor and mentor John Cantwell are those that diversify more technologically speaking over very long period of time so they are active in technologies that are not necessarily related to their production lines now um geographical analysis here is by country we we have a problem of significance of data uh recording in the case of taiwan and then china so there is a i mean the series starts to be uh reliable from late 90s but what we can see is that from here south korea japan and taiwan are those picking up the most particularly in the last couple of decades more interestingly is to look at to repeat the same exercise we did for uh the technologies but by big areas okay using the united nation macro region americas asia and europe now these are is the same exercise okay conditional means broken uh by introverts of five years over time to capture this relationship and we can see that i mean the three areas started from a comparable level in 1975 and then you can see that the americas okay became even significantly different from the other areas although interestingly while europe was going down and this obviously reflects somehow also specialization behind what what has grown particularly since the 2000 is asia and you can in ict and ai intersection technologies and you can see that asia is although is not significantly different from the americas but is on the trend let's say to become so so then we started to look at the specific geography of all this right and these are simple maps okay the maps using ArcMap or whatever software in order to identify the location of the most productive assignees and so we we we tried with different threshold uh in in the case of us and europe a good one was those assignees so those firms that have more than 100 patents and in the case of asia we had to put more than 500 patents here i report these maps and you can see that it's interesting because the geography of this most productive patent holder that are related to this critical materials are obviously in in specific locations where in fact there is this preponderance of multinational corporation big tech and clusters okay so i mean it's california uh new york state and and texas uh in the case of the us in europe you can see that is mostly north of europe than london a lot germany and also the netherlands belgium but very very specific cluster so these firms obviously interest some location in asia we see that is obviously a tokyo the top location because tokyo and osaka actually because osaka also have uh one i mean it's one of the hubs but followed by taipei singapore and beijing in china so then in a subsequent step we are still working on that this is very rough uh this is firm patenting versus each side it's just one big map so you cannot really see anything because it's difficult to to uh to disentangle all the six minerals but basically the message that we have in more detail maps is that you know as expected the technology is all global north and most not all but most of the critical and conflict mineral particularly the conflict actually subset are in the global south so this is the first snapshot of the idea and what we can conclude is that there is evidence although imperfect although we are we are you know uh um thinking about many possible ways to think about this association between ict technological paradigm and this conflict mineral innovation uh first of all we want to expanding other sets of technologies that are not ict but also to refine a bit better this method in order to clean it for example by uh i mean there might be patents that are now relying on on recycling these materials that can be confused with ours i mean we are aware of a limit a series of limitations on this however there is this association and also data validation production data i i forgot to say that the the the the mining sites are comes from obviously a different data set that is is uh based on the us geological survey um and so um there are there are different data sources to match also in order to validate better this methodology um ict is significantly different from other technologies in terms of their reference to critical and conflict materials we see that um there are two sort of forms of special disparities or two levels of special special inequality if you want that is the the expected one between uh uh global south and global north particularly and and here there is also the attempt now with with another quarter to capture better uh uh the some of these critical or conflict materials because then there are the rare earths and you know so there are there is there is scope of of increasing the the the analysis also from the material point of view understanding better the material for which i i as i told you we need support from colleagues in in in different fields um and also to explore for example in the in the in the logic of the value chain which are those that could in principle create generate some kind of more positive spillovers for the local uh for the local extraction industry locally owned or miners and then there is the second level of a special disparity that sees the concentration of this of this technological demand in some very few location hotspot of the global north okay that uh are the the the the the home of the big uh tech uh and uh and so this connects to another another you know other other lines of research that look precisely at the monopoly of the the large tech and their disruptive uh effects on on you know on global north but also on global south and so we we conclude by saying that you know this this intuition that specific technologies area paradigm have actually distinctive resource signature okay it is a relation that is interesting to explore and that has a lot of important implication in terms of demand of these resources and also geography of these resources so um it's in progress thank you thank you very much thank you very much simona um so if you want to stop sharing then uh exactly i was i was trying on the topic yes so that i can see the people yeah so we can all also give the floor to uh people that want to ask questions so uh john goddard i was asking john do you want to ask the question yourself you yes thank you hi there sir hey francis you were getting on to my question towards the end which was about the the industrial structure of the supply side and the power structures there and the companies and how the supply chain was functioning and the extent to which it was global north companies that were controlling the supply chain as well could you elaborate a bit on on on on that side of your analysis thank you uh yeah thank you very much that side is not in our analysis although we are obviously aware and we are this is part also of of the future of the research that has to really focus on the value chain we haven't we are we have started by focusing on technology okay it's it's a huge topic so it's uh it's difficult to to but yes uh mining companies and and the you know geographical origin of the mining company is is very very important matters a lot in particular matters because uh in the last decades there has been a search of chinese mining companies in particular the the context of africa or the triangle the the lithium triangle in in latin america so that is something that we will try to attach to this research and actually there is already work done on that and therefore we will try to connect if anyone is is actually has anything on that we will be very very interested in we cannot cover everything in this respect but is a very crucial point thank you thank you hey lisa do you want to ask yes thank you um nice to see you simona hi lisa um you look really well thank you so much and also i really enjoy your talk i have to say absolutely fascinating something that uh sometimes is forgotten what happens at the very very very beginning of a value chain and i was just wondering because i really have no idea what kind of innovation are done on these in relation to this ccm is that about mining is about processing um i just could not visualize what type of innovation would be patented in relation to these mineralism another question that i have if i may is is there any chance that some of these innovation so these patterns on on cmm so ccm can they can can they also be applied to other industries like i'm thinking for example automotive i'm thinking about i mean although autos are becoming more and more digital uh enable but still did you check or do you think is worth checking whether the ccm patents might actually be used in other sectors thank you very very interesting presentation thank you thank you thank you lisa thank you very much um uh yeah it's uh it's a very wide use and it's related to precisely batteries so we are now exploring precisely uh i mean with the with the with the geo geology friends they have a lot of industry let's say connections for uh uh with the with the automotive industry okay precisely because he's one of the big user so consider here we are looking at technologies not at industries but it's obviously one link to be done as as john was suggesting also you know this industry structure story okay uh these are minerals i mean the cobalt is crucial for miniaturization of electronic electronic components okay so every single mobile laptop has a cobalt inside okay so a macroelectronic circuit so it is a very wide range so we look we we had to select ict technologies but i mean the the use of these technologies at the industry level is wide okay so i'm talking about ict technologies not necessarily ict industries the user of the technology so i mean it's uh it's uh um it's a very very wide use because monetarization cannot happen without some of these materials okay and then the more you expand into other of this i mean for example the rare earth the more you find specific technologies that are specific to some industries but ict being a you know as we know general purpose eu it's used by a huge number of of industries automotive is one big user okay so uh this is again something that we we will uh we will uh do also because by look as you know better than me shifting to value chains you need industries so uh it's it's a bit problematic because for example it would be very interesting to and we started to look at export imports of this critical material which in some cases is very distorted because just to give you an example the cobalt that comes from the democratic republic of congo is actually smogged through rwanda and so there is a whole gray gray to not called dark area there that you know it's uh it's difficult it needs more thinking how to to capture there is work for everybody anyway here so join in i thank you very much thank you simona that was very thought provoking it's a bit outside my research area but it's it's really interesting i think it will be really i have three types of three kind of comments i think the first would be really interesting to see what size of economic activity depends on these materials and i mean now it will be approaching 100 i think indirectly at least for sure but it would be really interesting to see if there is some sort of indication of what we are talking about what is enabled by these materials the second is i'm not familiar with the data from the ustpo but my understanding limited is that you it's the most expensive patent office to register with and it's also kind of necessary if you're doing business in the us so you may have some self-selection in the geography between america's asia and europe and what you'll be observing probably is like the the european and the asian are the the real tigers of these countries that that register there probably and the final thing is it will be very interesting to to see if you can observe in the data at what level of demand or price or geopolitical reasons we have episodes that create substitute technologies so technologies that do not use cobalt anymore because cobalt becomes more expensive or is owned by a country that another country doesn't work want to work with et cetera but these are all nothing a bit too simple no no no no you you touch upon a very important point i mean size of the activity as i said okay i mean this is the first step in this research and so we had to start with something and we started with technology uspto you are right we are very well aware of the the bias however think about something first of all us pto has been the one that gives you more solid data back on time okay so it really covers from the 70s onwards and therefore to look at the trend is very very important it is also true that this kind of technological development is as our data set shows very clearly mostly done by big tech and big tech wherever they are they patent in all of these data sets now we can at some point replicate once we are more uh you know into the we can replicate with the you know the european patent office but one actually one interesting thing that is emerged by presenting this work has been also to look at the chinese patent data because chinese are very very big in this and therefore it might be that the number of smaller player appears so now i have a chinese student that is looking into that because obviously you know natural language processing in chinese is beyond the the competences that the research team has for the moment so that is important but you are right i mean we it's it's also in the paper this note on the uspto that might be slightly biased and substitute technologies that is actually what is in the conclusion uh critically uh highlighted as one of the final objective of this research is to look at the looking at this level of technology over time if we can capture something that departs and therefore that starts substituting technologies that uh not only technically substitute but technology that recycle better and you know there is a potential for some kind of prediction based on on this we have to refine a little bit more and then the method in order then to be able maybe to pick up something that is a bit more let's say positive and cheerful because here the story at the end of the day is not well i don't i don't see any more person um wanted to ask any question but i have some some comments uh so as uh lisa another say i think it was very very interesting i think it's a it's a noble way of always to think about something that many other researchers have have been looking at the past you know with all the um pharmaceutical patterns history and things like that in a more kind of novel and actual way um because uh i think nowadays uh big firms are so much concerned about also all the social corporate responsibility uh i'm i'm also thinking that um the social impact of those uh value chains if you want uh it's something uh worth to explore in particular because you're talking about those countries and where those patterns have created value you know to whom has benefited also thinking about maybe ownership and corporate governance you were talking about how chinese companies are exploiting those kind of mineral mines in using sometimes children so i think that that there is a huge scope in in thinking about the the social implications of of all of these value chains but um that on the on the on the origin uh maybe of this mineral side on the other side i think that there is also a new literature on on the value of patterns and also thinking not only on the quantities but also uh different patterns generate a different value added and also they have different type of value in where they are used so um i think it's linked with what lisa was saying if they are here some of them they might be more key enabled technologies other might be more specific uh so therefore maybe the use of one mineral or another might have a different purpose as the final value of that particular pattern um so um yeah i think browning hall is doing a lot on the value of patterns comparing to the quantity of patterns so there are some papers on that um i think somebody in the chat thank you raquel thank you very much these are all very important considerations in fact you saw in my very compacted one slide on the literature there is already huge literature that looks at this minerals and you know the link with the corporate social responsibility now i mean i i have my my own uh uh reservation let's say on the on the on the tail of corporate social responsibility okay because it's it's it's been in the sense it's been a way also to shift to the private business firm the the task of ensuring that everything is okay and it's been there you know also in the last let's say 20 years has become very big but i see other colleagues like elisa giuliani working a lot on that i know very well her work actually this research has been presented for the first time in a workshop organized by me and elisa on the dark side of the the the innovation of innovation at the german last year and i mean you know the work the serious work done on this shows that a lot is actually window dressing okay so um i i feel and i mean i'm not qualified to go into that but i feel that there is seriously the need to regulate this in a solid way i know that a lot of work is is is being carried out at the european commission level on this for example and in the united states of america regard regulatory bodies there is there are connection between these two areas in this respect and i see things moving more on that than telling firms be good also because think about some of these critical areas now the the democratic republic of congo is estimated although this again is you you you check different sources and you see different estimates but to produce between 35 and 60 percent of the cobalt in the world now talking about corporate social responsibility in firms that are located there right i mean just two days ago the italian ambassador was killed in there okay under the the the eye of everybody so i'm i'm not sure that is the way to go something can be done to strengthen that in certain contexts but i i am a bit uh uh skeptical uh on on on the on the real solution leaning on on on the corporate social responsibility uh the value of patented is an excellent suggestion uh because i believe that it's very relevant in this so browning hall i will i will check that is a very very good suggestion and i saw so that john was mentioning a sustainable development goal we we are in fact trying to align with that literature too i have another phd student who is actually focusing on on sdg uh particularly relative to africa focusing to link this topic with sdg because it's obviously very very important but as i said i mean wha what was a bit the um you know the the spirit is this is a big topic not we're not going to go into all the directions because we don't have the competences to do that okay there is all the the the the other side of the story is in in the in the exploitation of these minerals by local populations okay and that could could be really a very important source of local economic development and you know there you need also other tools other methodologies so it's it's really very open and it's just i wanted to flag if anyone thinks about doing this please let me know because we need in chronic then there is rampant clay i don't know if you know him and he does a lot on social impact assessment he has uh also uh every year winter and summer schools on social impact assessment with uh practitioners and so uh and they do a lot of work on africa different type of plantations and exploitation so i think this is probably one of the friends i can recommend you to go um i can introduce you to him and and the wife that also has a company on social impact assessment um yeah so um yeah fascinating uh i think that uh we are right on time i think uh so if there is no more questions um who can i ask a question yeah absolutely yes please thank you so much for simula's talk um i noticed you have a geographical analysis and also you have the temporal analysis about the change over time i just wondered whether you have considered it would be um any classroom pattern if you look at by the country boundary um in terms of the way the patent um emergence across the time because i would imagine even though you mentioned they are clustered in amsterdam and also some major cities in major countries but i mean the emergence of this patent ib actually occurred have any patent based on one country or the randomly appearance in different places mainly because of a company um i wonder whether that's one aspect that you have looked into and on the other hand you mentioned you'll be interested in explore the patent in china i thought i can offer you a thought about my experience with chinese patent um so basically the patent system in china and they were designed um in a different way in as incentive as uk system and u.s system i remember like researcher told me well they have to pay themselves to get patent and also as a way of being examined by the government and university as a way to get promoted so i guess in from that perspective patent has not only been as economic incentive for a researcher but also they have their own willingness to patent it so i would imagine that would have an impact that could possibly lead to certain um i wouldn't know how to explain that but i would think they are kitten for a different reason and just thought on that thank you no thank you very very much what you see about the location of obviously i mean we start with the big firms or we start with the big the big players okay because what we we didn't know we didn't know but i mean really the big majority is very very important that is all very concentrated at the organizational level and uh it's interesting because as you know patents do not register only firms okay there can be other organizations and um actually the only non private owner non-business owner big is the board of university of california that appears in the the top 500 okay interestingly enough so i mean you know we will go to and now i showed you the maps with the threshold because otherwise also in order to present the map you don't see anything so it's also a choice a bit dictated by visualization but we have to go much further with the geographical analysis this was really a preliminary story uh the chinese patents i i know uh the very big difference so using chinese patents won't obviously allow any comparison with other patent sources okay while tasers was saying epo data can give you a bit different picture and they can be roughly compared to your spto capturing chinese data and is not it's not in that direction i know i know here that we have a big problem of quality in fact however it's interesting because guess what looking at this preliminary uh the big patent owner in this uh in this critical and uh materials are not business firms are universities in the case of china so interesting yeah to to see yeah you will see at some point i mean we can we can also get in contact with this i would love to follow up thank you so much um i also remember you used to write about the structural agglomeration do you think you might go into it and explore the different kind of different type of agglomeration type driven by transaction code or that kind of structure to examine a little bit about it there are different directions right i mean it's it's it's a big job so we'll we'll i have to find also some money for it because you know is israel is rather uh is rather labor intensive let's put it this way so well thank you very much simona i think everybody enjoyed very much your presentation so it looks very very very promising so i think i think we are all of us we are thinking in what is coming next so uh keep working on that thank you thank you so much thank you all for uh for a very very nice exchange and as said if any of you is doing something related because it's part of a big puzzle and it would be really nice to share thank you really very much it was a pleasure and thank you very much all for uh joining us today uh in two weeks time we will have uh kevin morgan also uh giving a seminar so um we look forward for the next seminar as well um have a nice day and see you soon bye bye bye bye thank you all