Since you're building an interaction that's more complex than simply creating a point where the user clicked, I suggest that you use a button.
You need to define exactly what action you want to implement: remove the last point, remove all the points, remove all the points but the ones that were used for initialization...
I'm gonna show you how you can create a Reset button that will remove all the points, based upon this example from Matplotlib's doc.
First, create an Axes object that your button will fill.
You'll want to adjust the main ax so the two don't overlap.
from matplotlib.widgets import Button
plt.subplots_adjust(bottom=0.2)
breset_ax = plt.axes([0.7, 0.05, 0.1, 0.075])
breset = Button(breset_ax, 'Reset')
Then you'll set the button's callback.
I find it relevant to define that callback in the LineBuilder class, as it will clear the encapsulated points.
class LineBuilder:
...
def reset(self, _event):
self.xs = []
self.ys = []
self.line.set_data(self.xs, self.ys)
self.line.figure.canvas.draw_idle()
Then, bind that callback to the button:
breset.on_clicked(linebuilder.reset)
This will give you something like that:

And clicking on the Reset button will remove all the points that have been drawn.
Complete code:
from matplotlib import pyplot as plt
from matplotlib.widgets import Button
class LineBuilder:
def __init__(self, line):
self.line = line
self.xs = list(line.get_xdata())
self.ys = list(line.get_ydata())
self.cid = line.figure.canvas.mpl_connect('button_press_event', self)
def __call__(self, event):
if event.inaxes!=self.line.axes:
return
self.xs.append(event.xdata)
self.ys.append(event.ydata)
self.line.set_data(self.xs, self.ys)
self.line.figure.canvas.draw_idle()
print(self.xs)
print(self.ys)
def reset(self, _event):
self.xs = []
self.ys = []
self.line.set_data(self.xs, self.ys)
self.line.figure.canvas.draw_idle()
fig = plt.figure()
ax = fig.add_subplot(111)
ax.set_title('click to build line segments')
line, = ax.plot([0], [0], marker="o", linestyle="")
linebuilder = LineBuilder(line)
plt.subplots_adjust(bottom=0.2)
breset_ax = plt.axes([0.7, 0.05, 0.1, 0.075])
breset = Button(breset_ax, 'Reset')
breset.on_clicked(linebuilder.reset)
plt.show()