Of course, it could. It could also be worse. This world has already been touched by us. Sometimes little things make huge differences. As I said Jarvis has information he did not have before that means Ultron will have other information. You do not know where it will end.
I have heard about worlds where we have been to twice. A door opens one time, some go through and months or years later, a door opens to the same world and people remembered us and things were different. In some they were vastly different. It might be one life changed, it might be plenty.
What I suggest is to not look at this world as a past, this planet cannot be your past. It is a present. All I am suggesting is to give it a nudge in the right direction. Use the knowledge you have obtained. That's what people in the present do. That's what the Avengers did.